| 
				
			 | 
			
			
				@@ -1,360 +0,0 @@ 
			 | 
		
	
		
			
			| 
				1
			 | 
			
				
			 | 
			
			
				-Tests 
			 | 
		
	
		
			
			| 
				2
			 | 
			
				
			 | 
			
			
				-===== 
			 | 
		
	
		
			
			| 
				3
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				4
			 | 
			
				
			 | 
			
			
				-Running tests is handled by tfkit/runtests: 
			 | 
		
	
		
			
			| 
				5
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				6
			 | 
			
				
			 | 
			
			
				-    $ tfkit/runtest [filter] 
			 | 
		
	
		
			
			| 
				7
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				8
			 | 
			
				
			 | 
			
			
				-*filter* is a regular expression to be applied to sub-test name, running 
			 | 
		
	
		
			
			| 
				9
			 | 
			
				
			 | 
			
			
				-only the matching ones.  See below for details. 
			 | 
		
	
		
			
			| 
				10
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				11
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				12
			 | 
			
				
			 | 
			
			
				-Writing tests 
			 | 
		
	
		
			
			| 
				13
			 | 
			
				
			 | 
			
			
				-------------- 
			 | 
		
	
		
			
			| 
				14
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				15
			 | 
			
				
			 | 
			
			
				-Tests can be written in any scripting language, although the built-in 
			 | 
		
	
		
			
			| 
				16
			 | 
			
				
			 | 
			
			
				-framework, written in Bash, provides some useful features for writing 
			 | 
		
	
		
			
			| 
				17
			 | 
			
				
			 | 
			
			
				-certain kind of relatively simple tests. 
			 | 
		
	
		
			
			| 
				18
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				19
			 | 
			
				
			 | 
			
			
				-The harness, though, assumes that: 
			 | 
		
	
		
			
			| 
				20
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				21
			 | 
			
				
			 | 
			
			
				- *  Any direct sub-directory of `$TF_SUITE` directory ("tests" by default) 
			 | 
		
	
		
			
			| 
				22
			 | 
			
				
			 | 
			
			
				-    that contains at least *TF_RUN* executable becomes a test, 
			 | 
		
	
		
			
			| 
				23
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				24
			 | 
			
				
			 | 
			
			
				- *  basename of this directory becomes the name of the test, 
			 | 
		
	
		
			
			| 
				25
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				26
			 | 
			
				
			 | 
			
			
				- *  and return code from running the executable is reported 
			 | 
		
	
		
			
			| 
				27
			 | 
			
				
			 | 
			
			
				-    as result of the test, according to "Exit status" chapter below. 
			 | 
		
	
		
			
			| 
				28
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				29
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				30
			 | 
			
				
			 | 
			
			
				-Naming 
			 | 
		
	
		
			
			| 
				31
			 | 
			
				
			 | 
			
			
				------- 
			 | 
		
	
		
			
			| 
				32
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				33
			 | 
			
				
			 | 
			
			
				-Test name should start with name of the module that is tested and 
			 | 
		
	
		
			
			| 
				34
			 | 
			
				
			 | 
			
			
				-underscore.  If module name contains dots, they should be replaced with 
			 | 
		
	
		
			
			| 
				35
			 | 
			
				
			 | 
			
			
				-underscores as well. 
			 | 
		
	
		
			
			| 
				36
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				37
			 | 
			
				
			 | 
			
			
				-    core_sanity 
			 | 
		
	
		
			
			| 
				38
			 | 
			
				
			 | 
			
			
				-    mod_submod_function 
			 | 
		
	
		
			
			| 
				39
			 | 
			
				
			 | 
			
			
				-    ini_iniread 
			 | 
		
	
		
			
			| 
				40
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				41
			 | 
			
				
			 | 
			
			
				-are valid test names. 
			 | 
		
	
		
			
			| 
				42
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				43
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				44
			 | 
			
				
			 | 
			
			
				-Data 
			 | 
		
	
		
			
			| 
				45
			 | 
			
				
			 | 
			
			
				----- 
			 | 
		
	
		
			
			| 
				46
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				47
			 | 
			
				
			 | 
			
			
				-Should the test need any data, just leave it around in the test directory 
			 | 
		
	
		
			
			| 
				48
			 | 
			
				
			 | 
			
			
				-along with *TF_RUN*. 
			 | 
		
	
		
			
			| 
				49
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				50
			 | 
			
				
			 | 
			
			
				-Note that before running, the whole test directory is automatically 
			 | 
		
	
		
			
			| 
				51
			 | 
			
				
			 | 
			
			
				-copied to a temporary location (one per test), and should the test fail, 
			 | 
		
	
		
			
			| 
				52
			 | 
			
				
			 | 
			
			
				-copied back as a debugging artifact.  For this reason, *do not store 
			 | 
		
	
		
			
			| 
				53
			 | 
			
				
			 | 
			
			
				-huge amounts of data here*.  If you really need huge data, consider 
			 | 
		
	
		
			
			| 
				54
			 | 
			
				
			 | 
			
			
				-obtaining it (and throwing it away) within runtime of *TF_RUN*. 
			 | 
		
	
		
			
			| 
				55
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				56
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				57
			 | 
			
				
			 | 
			
			
				-Exit status 
			 | 
		
	
		
			
			| 
				58
			 | 
			
				
			 | 
			
			
				------------ 
			 | 
		
	
		
			
			| 
				59
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				60
			 | 
			
				
			 | 
			
			
				-We try hard to follow this semantic: 
			 | 
		
	
		
			
			| 
				61
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				62
			 | 
			
				
			 | 
			
			
				- *  *Zero* means *OK* -- test has been run and passed. 
			 | 
		
	
		
			
			| 
				63
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				64
			 | 
			
				
			 | 
			
			
				- *  *One* means *Failure* -- test has been run but failed (e.g. found 
			 | 
		
	
		
			
			| 
				65
			 | 
			
				
			 | 
			
			
				-     a bug). 
			 | 
		
	
		
			
			| 
				66
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				67
			 | 
			
				
			 | 
			
			
				- *  *Two* means *Bailout* --  test has decided not to run at all. 
			 | 
		
	
		
			
			| 
				68
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				69
			 | 
			
				
			 | 
			
			
				- *  *Three* means *Error* -- there was error detected during execution, 
			 | 
		
	
		
			
			| 
				70
			 | 
			
				
			 | 
			
			
				-     but script was able to clean up properly. 
			 | 
		
	
		
			
			| 
				71
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				72
			 | 
			
				
			 | 
			
			
				- *  *Four* means *Panic* -- there was other error but script *was not* 
			 | 
		
	
		
			
			| 
				73
			 | 
			
				
			 | 
			
			
				-     able to clean up properly. 
			 | 
		
	
		
			
			| 
				74
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				75
			 | 
			
				
			 | 
			
			
				- *  Anything else should indicate other uncaught errors, including those 
			 | 
		
	
		
			
			| 
				76
			 | 
			
				
			 | 
			
			
				-    outside control of the program such as segfaults in the test code 
			 | 
		
	
		
			
			| 
				77
			 | 
			
				
			 | 
			
			
				-    or test being SIGKILLed. 
			 | 
		
	
		
			
			| 
				78
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				79
			 | 
			
				
			 | 
			
			
				-Notice that the higher the value is, the worse situation it indicates. 
			 | 
		
	
		
			
			| 
				80
			 | 
			
				
			 | 
			
			
				-Thus, if a test is composed of several sub-tests, you need to make sure 
			 | 
		
	
		
			
			| 
				81
			 | 
			
				
			 | 
			
			
				-to always **exit with the highest value** (subtest.sh does take care 
			 | 
		
	
		
			
			| 
				82
			 | 
			
				
			 | 
			
			
				-of this). 
			 | 
		
	
		
			
			| 
				83
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				84
			 | 
			
				
			 | 
			
			
				-See *common.sh* for functions and variables to help with handling exit 
			 | 
		
	
		
			
			| 
				85
			 | 
			
				
			 | 
			
			
				-statuses with this semantic. 
			 | 
		
	
		
			
			| 
				86
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				87
			 | 
			
				
			 | 
			
			
				-Also see Notes section for more details on exit statuses, including 
			 | 
		
	
		
			
			| 
				88
			 | 
			
				
			 | 
			
			
				-cheat sheet and dscussuion. 
			 | 
		
	
		
			
			| 
				89
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				90
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				91
			 | 
			
				
			 | 
			
			
				-Framework 
			 | 
		
	
		
			
			| 
				92
			 | 
			
				
			 | 
			
			
				---------- 
			 | 
		
	
		
			
			| 
				93
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				94
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				95
			 | 
			
				
			 | 
			
			
				-### harness.sh ### 
			 | 
		
	
		
			
			| 
				96
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				97
			 | 
			
				
			 | 
			
			
				-This part is not intended to be used in tests, but rather contains 
			 | 
		
	
		
			
			| 
				98
			 | 
			
				
			 | 
			
			
				-functions that help govern test discovery, preparation and execution as 
			 | 
		
	
		
			
			| 
				99
			 | 
			
				
			 | 
			
			
				-is described in previous chapters.  Feel free to poke around, of course. 
			 | 
		
	
		
			
			| 
				100
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				101
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				102
			 | 
			
				
			 | 
			
			
				-### subtest.sh ### 
			 | 
		
	
		
			
			| 
				103
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				104
			 | 
			
				
			 | 
			
			
				-As name suggests, this file defines few functions to handle subtests 
			 | 
		
	
		
			
			| 
				105
			 | 
			
				
			 | 
			
			
				-in *TF_RUN*. 
			 | 
		
	
		
			
			| 
				106
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				107
			 | 
			
				
			 | 
			
			
				-In order to make use of the subtests functionality, you will need to 
			 | 
		
	
		
			
			| 
				108
			 | 
			
				
			 | 
			
			
				-define two functions yourself:  `tf_enum_subtests` to enumerate names of 
			 | 
		
	
		
			
			| 
				109
			 | 
			
				
			 | 
			
			
				-tests you want to run, and `tf_name2cmd` to translate each name an actual 
			 | 
		
	
		
			
			| 
				110
			 | 
			
				
			 | 
			
			
				-command that would perform it and return with the the correct exit status. 
			 | 
		
	
		
			
			| 
				111
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				112
			 | 
			
				
			 | 
			
			
				-The minimal *TF_RUN* with two subtests could look like this: 
			 | 
		
	
		
			
			| 
				113
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				114
			 | 
			
				
			 | 
			
			
				-    #!/bin/bash 
			 | 
		
	
		
			
			| 
				115
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				116
			 | 
			
				
			 | 
			
			
				-    . $TF_DIR/include/subtest.sh 
			 | 
		
	
		
			
			| 
				117
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				118
			 | 
			
				
			 | 
			
			
				-    tf_enum_subtests() { 
			 | 
		
	
		
			
			| 
				119
			 | 
			
				
			 | 
			
			
				-        echo test1 
			 | 
		
	
		
			
			| 
				120
			 | 
			
				
			 | 
			
			
				-        echo test2 
			 | 
		
	
		
			
			| 
				121
			 | 
			
				
			 | 
			
			
				-        something && echo test3 
			 | 
		
	
		
			
			| 
				122
			 | 
			
				
			 | 
			
			
				-    } 
			 | 
		
	
		
			
			| 
				123
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				124
			 | 
			
				
			 | 
			
			
				-    tf_name2cmd() { 
			 | 
		
	
		
			
			| 
				125
			 | 
			
				
			 | 
			
			
				-        case $1 in 
			 | 
		
	
		
			
			| 
				126
			 | 
			
				
			 | 
			
			
				-            test1)  echo myprog foo ;; 
			 | 
		
	
		
			
			| 
				127
			 | 
			
				
			 | 
			
			
				-            test2)  echo myprog bar ;; 
			 | 
		
	
		
			
			| 
				128
			 | 
			
				
			 | 
			
			
				-        esac 
			 | 
		
	
		
			
			| 
				129
			 | 
			
				
			 | 
			
			
				-    } 
			 | 
		
	
		
			
			| 
				130
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				131
			 | 
			
				
			 | 
			
			
				-    tf_do_subtests 
			 | 
		
	
		
			
			| 
				132
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				133
			 | 
			
				
			 | 
			
			
				-At the end, `tf_do_subtests` acts as a launcher of the actual test. 
			 | 
		
	
		
			
			| 
				134
			 | 
			
				
			 | 
			
			
				-In short, it will 
			 | 
		
	
		
			
			| 
				135
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				136
			 | 
			
				
			 | 
			
			
				- *  take each enumerated subtest from `tf_enum_subtests`, 
			 | 
		
	
		
			
			| 
				137
			 | 
			
				
			 | 
			
			
				- *  source *TF_SETUP*, if such file is found, 
			 | 
		
	
		
			
			| 
				138
			 | 
			
				
			 | 
			
			
				- *  translate te subtest name to a command, 
			 | 
		
	
		
			
			| 
				139
			 | 
			
				
			 | 
			
			
				- *  launch the command, 
			 | 
		
	
		
			
			| 
				140
			 | 
			
				
			 | 
			
			
				- *  source *TF_CLEANUP*, if such file is found, 
			 | 
		
	
		
			
			| 
				141
			 | 
			
				
			 | 
			
			
				- *  and report "worst" exit status encountered. 
			 | 
		
	
		
			
			| 
				142
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				143
			 | 
			
				
			 | 
			
			
				-All but the first and last step is done by `tf_do_subtest`, so in some 
			 | 
		
	
		
			
			| 
				144
			 | 
			
				
			 | 
			
			
				-cases you may want to re-define this one as well. 
			 | 
		
	
		
			
			| 
				145
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				146
			 | 
			
				
			 | 
			
			
				-Note that subtest names need to be single words (`[a-zA-Z0-9_]`). 
			 | 
		
	
		
			
			| 
				147
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				148
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				149
			 | 
			
				
			 | 
			
			
				-### tools.sh ### 
			 | 
		
	
		
			
			| 
				150
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				151
			 | 
			
				
			 | 
			
			
				-This file contains various tools and utilities to help with testing. 
			 | 
		
	
		
			
			| 
				152
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				153
			 | 
			
				
			 | 
			
			
				-Curently there is only one function, `tf_testflt` designed to help write 
			 | 
		
	
		
			
			| 
				154
			 | 
			
				
			 | 
			
			
				-tests for simple unix filters. 
			 | 
		
	
		
			
			| 
				155
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				156
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				157
			 | 
			
				
			 | 
			
			
				-#### tf_testflt #### 
			 | 
		
	
		
			
			| 
				158
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				159
			 | 
			
				
			 | 
			
			
				-The idea is that tester specifies 
			 | 
		
	
		
			
			| 
				160
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				161
			 | 
			
				
			 | 
			
			
				- *  test name, 
			 | 
		
	
		
			
			| 
				162
			 | 
			
				
			 | 
			
			
				- *  command to launch the system under test, 
			 | 
		
	
		
			
			| 
				163
			 | 
			
				
			 | 
			
			
				- *  a data stream to use as STDIN, 
			 | 
		
	
		
			
			| 
				164
			 | 
			
				
			 | 
			
			
				- *  and expected STDOUT, STDERR, and exit status. 
			 | 
		
	
		
			
			| 
				165
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				166
			 | 
			
				
			 | 
			
			
				-and tf_testflt launches the command, collects tha data and evaluates 
			 | 
		
	
		
			
			| 
				167
			 | 
			
				
			 | 
			
			
				-and reports the result using unified diff. 
			 | 
		
	
		
			
			| 
				168
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				169
			 | 
			
				
			 | 
			
			
				-In its simplest form: 
			 | 
		
	
		
			
			| 
				170
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				171
			 | 
			
				
			 | 
			
			
				-    tf_testflt -n foo my_command arg 
			 | 
		
	
		
			
			| 
				172
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				173
			 | 
			
				
			 | 
			
			
				-the function will run `my_command arg` (not piping anything to it), 
			 | 
		
	
		
			
			| 
				174
			 | 
			
				
			 | 
			
			
				-and will expect it to finish with exit status 0 and empty both STDERR 
			 | 
		
	
		
			
			| 
				175
			 | 
			
				
			 | 
			
			
				-and STDOUT. 
			 | 
		
	
		
			
			| 
				176
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				177
			 | 
			
				
			 | 
			
			
				-Example of full form, 
			 | 
		
	
		
			
			| 
				178
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				179
			 | 
			
				
			 | 
			
			
				-    tf_testflt -n foo -i foo.in -O foo.stdout -E foo.stderr -S 2 myprog 
			 | 
		
	
		
			
			| 
				180
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				181
			 | 
			
				
			 | 
			
			
				-will pipe foo.in into `myprog`, expecting exit status of 2, and STDOUT and 
			 | 
		
	
		
			
			| 
				182
			 | 
			
				
			 | 
			
			
				-STDERR as above.  Notice that parameters specifying expected values are 
			 | 
		
	
		
			
			| 
				183
			 | 
			
				
			 | 
			
			
				-uppercase, and those specifying input values are lowercase. 
			 | 
		
	
		
			
			| 
				184
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				185
			 | 
			
				
			 | 
			
			
				-Specifying name is mandatory, because it's used in reporting messages, 
			 | 
		
	
		
			
			| 
				186
			 | 
			
				
			 | 
			
			
				-and as a basis for naming temporary result files: these are saved in 
			 | 
		
	
		
			
			| 
				187
			 | 
			
				
			 | 
			
			
				-*results* subdirectory and kept for further reference. 
			 | 
		
	
		
			
			| 
				188
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				189
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				190
			 | 
			
				
			 | 
			
			
				-### common.sh ### 
			 | 
		
	
		
			
			| 
				191
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				192
			 | 
			
				
			 | 
			
			
				-This includes simple functions and variables shared between both mentioned 
			 | 
		
	
		
			
			| 
				193
			 | 
			
				
			 | 
			
			
				-libraries. 
			 | 
		
	
		
			
			| 
				194
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				195
			 | 
			
				
			 | 
			
			
				-First group is designed to help support the exit status semantic: 
			 | 
		
	
		
			
			| 
				196
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				197
			 | 
			
				
			 | 
			
			
				- *  The functions are `tf_exit_pass`, `tf_exit_fail`, `tf_exit_bailout`, 
			 | 
		
	
		
			
			| 
				198
			 | 
			
				
			 | 
			
			
				-    `tf_exit_error` and `tf_exit_panic` and each take any number of 
			 | 
		
	
		
			
			| 
				199
			 | 
			
				
			 | 
			
			
				-    parameters that are printed on stderr. 
			 | 
		
	
		
			
			| 
				200
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				201
			 | 
			
				
			 | 
			
			
				- *  The variables are `TF_ES_OK`, `TF_ES_FAIL`, `TF_ES_BAILOUT`, 
			 | 
		
	
		
			
			| 
				202
			 | 
			
				
			 | 
			
			
				-    `TF_ES_ERROR` and `TF_ES_PANIC` and are supposed to be used with 
			 | 
		
	
		
			
			| 
				203
			 | 
			
				
			 | 
			
			
				-    `return` builtin, e.g. to return from `tf_exit_error`. 
			 | 
		
	
		
			
			| 
				204
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				205
			 | 
			
				
			 | 
			
			
				-Second group is useful to better control output:  functions `tf_warn`, 
			 | 
		
	
		
			
			| 
				206
			 | 
			
				
			 | 
			
			
				-`tf_debug` and `tf_think` are used to print stuff on STDERR.  Use of 
			 | 
		
	
		
			
			| 
				207
			 | 
			
				
			 | 
			
			
				-`tf_warn` is apparent, just as `tf_debug`, the latter being muted if 
			 | 
		
	
		
			
			| 
				208
			 | 
			
				
			 | 
			
			
				-`TF_DEBUG` is set to `false` (set it to `true` to turn on debugging). 
			 | 
		
	
		
			
			| 
				209
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				210
			 | 
			
				
			 | 
			
			
				-`tf_think` is used for progress info, and is muted unless `TF_VERBOSE` 
			 | 
		
	
		
			
			| 
				211
			 | 
			
				
			 | 
			
			
				-is set to `true`, which is by default. 
			 | 
		
	
		
			
			| 
				212
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				213
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				214
			 | 
			
				
			 | 
			
			
				-### Setup and cleanup ### 
			 | 
		
	
		
			
			| 
				215
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				216
			 | 
			
				
			 | 
			
			
				-Special files *TF_SETUP* and *TF_CLEANUP* (one of them or both) can be 
			 | 
		
	
		
			
			| 
				217
			 | 
			
				
			 | 
			
			
				-added along with *TF_RUN*.  These will be sourced before (*TF_SETUP*) 
			 | 
		
	
		
			
			| 
				218
			 | 
			
				
			 | 
			
			
				-and after every subtest (*TF_CLEANUP*). 
			 | 
		
	
		
			
			| 
				219
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				220
			 | 
			
				
			 | 
			
			
				-First, if any of these files are missing, it is considered as if the 
			 | 
		
	
		
			
			| 
				221
			 | 
			
				
			 | 
			
			
				-respective phase succeeded.  Second, if setup phase fails, test will 
			 | 
		
	
		
			
			| 
				222
			 | 
			
				
			 | 
			
			
				-be skipped and subtest exit status will be *TF_ES_BAILOUT*.   Last, 
			 | 
		
	
		
			
			| 
				223
			 | 
			
				
			 | 
			
			
				-if cleanup fails (no matter result of setup), subtests aborts with 
			 | 
		
	
		
			
			| 
				224
			 | 
			
				
			 | 
			
			
				-*TF_ES_PANIC* returned.  Be aware that in this case the actual test 
			 | 
		
	
		
			
			| 
				225
			 | 
			
				
			 | 
			
			
				-status, albeit useful, is lost. 
			 | 
		
	
		
			
			| 
				226
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				227
			 | 
			
				
			 | 
			
			
				-When coming from other test frameworks, this may feel harsh, but note 
			 | 
		
	
		
			
			| 
				228
			 | 
			
				
			 | 
			
			
				-that this has been designed with the idea that if a cleanup fails, 
			 | 
		
	
		
			
			| 
				229
			 | 
			
				
			 | 
			
			
				-it may render all further tests are automatically unsafe, because the 
			 | 
		
	
		
			
			| 
				230
			 | 
			
				
			 | 
			
			
				-environment is not as expected. 
			 | 
		
	
		
			
			| 
				231
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				232
			 | 
			
				
			 | 
			
			
				-To cope with this behavior, try to bear in mind following advice: 
			 | 
		
	
		
			
			| 
				233
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				234
			 | 
			
				
			 | 
			
			
				- 1. Make sure you write setup/cleanup procedures with extreme care and 
			 | 
		
	
		
			
			| 
				235
			 | 
			
				
			 | 
			
			
				-    test them well. 
			 | 
		
	
		
			
			| 
				236
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				237
			 | 
			
				
			 | 
			
			
				- 2. Do not do complicated and risky things in the setup/cleanup phases. 
			 | 
		
	
		
			
			| 
				238
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				239
			 | 
			
				
			 | 
			
			
				- 3. If you need to do such things, consider doing them in the *TF_RUN* 
			 | 
		
	
		
			
			| 
				240
			 | 
			
				
			 | 
			
			
				-    instead of doing them for all subtests. 
			 | 
		
	
		
			
			| 
				241
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				242
			 | 
			
				
			 | 
			
			
				- 4. You don't need to clean up everything, the contents of the testing dir 
			 | 
		
	
		
			
			| 
				243
			 | 
			
				
			 | 
			
			
				-    will be moved out from the test system. 
			 | 
		
	
		
			
			| 
				244
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				245
			 | 
			
				
			 | 
			
			
				- 5. If there are scenarios you can safely fix or ignore, handle them in 
			 | 
		
	
		
			
			| 
				246
			 | 
			
				
			 | 
			
			
				-    a robust manner. 
			 | 
		
	
		
			
			| 
				247
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				248
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				249
			 | 
			
				
			 | 
			
			
				-Notes 
			 | 
		
	
		
			
			| 
				250
			 | 
			
				
			 | 
			
			
				------ 
			 | 
		
	
		
			
			| 
				251
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				252
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				253
			 | 
			
				
			 | 
			
			
				-### bailout vs. `tf_enum_subtests` ### 
			 | 
		
	
		
			
			| 
				254
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				255
			 | 
			
				
			 | 
			
			
				-One more note to claify relation of bailout and `tf_enum_subtests`. 
			 | 
		
	
		
			
			| 
				256
			 | 
			
				
			 | 
			
			
				-As you may have noticed, there are two ways how to skip a test: 
			 | 
		
	
		
			
			| 
				257
			 | 
			
				
			 | 
			
			
				-return prematurely with `TF_ES_BAILOUT`, or suppress enumeration in 
			 | 
		
	
		
			
			| 
				258
			 | 
			
				
			 | 
			
			
				-`tf_enum_subtests`.  The problem is that the latter does not do anything 
			 | 
		
	
		
			
			| 
				259
			 | 
			
				
			 | 
			
			
				-to inform upper in the stack that a test has been skipped, which seems to 
			 | 
		
	
		
			
			| 
				260
			 | 
			
				
			 | 
			
			
				-break the principle described in the previous chapters. 
			 | 
		
	
		
			
			| 
				261
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				262
			 | 
			
				
			 | 
			
			
				-Don't confuse these mechanisms, though. Each is supposed to be used 
			 | 
		
	
		
			
			| 
				263
			 | 
			
				
			 | 
			
			
				-for distinct purpose.  Compare: by using the `tf_enum_subtests` you are 
			 | 
		
	
		
			
			| 
				264
			 | 
			
				
			 | 
			
			
				-saying that you actually **did not even want** to run the test in the 
			 | 
		
	
		
			
			| 
				265
			 | 
			
				
			 | 
			
			
				-first place.  By using `TF_ES_BAILOUT`, you are saying that you **wanted** 
			 | 
		
	
		
			
			| 
				266
			 | 
			
				
			 | 
			
			
				-to run the test but could not. 
			 | 
		
	
		
			
			| 
				267
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				268
			 | 
			
				
			 | 
			
			
				-A few common cases if that helps you: 
			 | 
		
	
		
			
			| 
				269
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				270
			 | 
			
				
			 | 
			
			
				- *  If during the test you find out that for some reason it can't be 
			 | 
		
	
		
			
			| 
				271
			 | 
			
				
			 | 
			
			
				-    carried out (e.g. an external resource is not available, or 
			 | 
		
	
		
			
			| 
				272
			 | 
			
				
			 | 
			
			
				-    something outside the SUT is broken), use `TF_ES_BAILOUT`. 
			 | 
		
	
		
			
			| 
				273
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				274
			 | 
			
				
			 | 
			
			
				- *  If you want to disable the test because for some long-term condition, 
			 | 
		
	
		
			
			| 
				275
			 | 
			
				
			 | 
			
			
				-    e.g. a known bug outside SUT but preventing execution of the test 
			 | 
		
	
		
			
			| 
				276
			 | 
			
				
			 | 
			
			
				-    is not fixed, use `tf_enum_subtests`. 
			 | 
		
	
		
			
			| 
				277
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				278
			 | 
			
				
			 | 
			
			
				- *  If you want to filter out some sub-tests to only for some platforms, 
			 | 
		
	
		
			
			| 
				279
			 | 
			
				
			 | 
			
			
				-    e.g. 64-bit architecture, (IOW, you can safely check that a 
			 | 
		
	
		
			
			| 
				280
			 | 
			
				
			 | 
			
			
				-    sub-test would be totally pointless if run on this box), use 
			 | 
		
	
		
			
			| 
				281
			 | 
			
				
			 | 
			
			
				-    `tf_enum_subtests`. 
			 | 
		
	
		
			
			| 
				282
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				283
			 | 
			
				
			 | 
			
			
				- *  If you want to disable (comment out test) that you might not have 
			 | 
		
	
		
			
			| 
				284
			 | 
			
				
			 | 
			
			
				-    implemented yet or is broken (and for some reason you still want 
			 | 
		
	
		
			
			| 
				285
			 | 
			
				
			 | 
			
			
				-    it to haunt the test code), use `tf_enum_subtests` and properly 
			 | 
		
	
		
			
			| 
				286
			 | 
			
				
			 | 
			
			
				-    comment the reasons in code. 
			 | 
		
	
		
			
			| 
				287
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				288
			 | 
			
				
			 | 
			
			
				- *  If in doubt, use `TF_ES_BAILOUT`. 
			 | 
		
	
		
			
			| 
				289
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				290
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				291
			 | 
			
				
			 | 
			
			
				-### On exit statuses: three and above ### 
			 | 
		
	
		
			
			| 
				292
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				293
			 | 
			
				
			 | 
			
			
				-The difference in *error*, *panic* and higher values is subtle but 
			 | 
		
	
		
			
			| 
				294
			 | 
			
				
			 | 
			
			
				-important.  Follow me as I try to explain: 
			 | 
		
	
		
			
			| 
				295
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				296
			 | 
			
				
			 | 
			
			
				- 1. If script has changed something on the system outside the working 
			 | 
		
	
		
			
			| 
				297
			 | 
			
				
			 | 
			
			
				-    directory, it is apparently expected to revert that change. 
			 | 
		
	
		
			
			| 
				298
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				299
			 | 
			
				
			 | 
			
			
				- 2. Now if an error occurs, but the code responsible for cleaning up is 
			 | 
		
	
		
			
			| 
				300
			 | 
			
				
			 | 
			
			
				-    safely run, you can say there was *error but we have recovered*. 
			 | 
		
	
		
			
			| 
				301
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				302
			 | 
			
				
			 | 
			
			
				- 3. But if the change can't be reverted safely, we know that we have 
			 | 
		
	
		
			
			| 
				303
			 | 
			
				
			 | 
			
			
				-    broken something and latter code may lead to weird results (including 
			 | 
		
	
		
			
			| 
				304
			 | 
			
				
			 | 
			
			
				-    masking bugs(!)), it's time to *panic* (in the code, not in real 
			 | 
		
	
		
			
			| 
				305
			 | 
			
				
			 | 
			
			
				-    life ;)) 
			 | 
		
	
		
			
			| 
				306
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				307
			 | 
			
				
			 | 
			
			
				- 4. And then there are corner cases like a bug in the script, OOM kill 
			 | 
		
	
		
			
			| 
				308
			 | 
			
				
			 | 
			
			
				-    or timeout when the status will be different and not really controlled 
			 | 
		
	
		
			
			| 
				309
			 | 
			
				
			 | 
			
			
				-    by the script.  Such cases will have to be treated the same way as 
			 | 
		
	
		
			
			| 
				310
			 | 
			
				
			 | 
			
			
				-    the "panic" case, but... 
			 | 
		
	
		
			
			| 
				311
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				312
			 | 
			
				
			 | 
			
			
				- 5. the use of *panic* adds hint that the status has been set consciously 
			 | 
		
	
		
			
			| 
				313
			 | 
			
				
			 | 
			
			
				-    by the script, albeit exiting "in a hurry"--without proper clean up. 
			 | 
		
	
		
			
			| 
				314
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				315
			 | 
			
				
			 | 
			
			
				-Unfortunately there will be cases like above but with the error code less 
			 | 
		
	
		
			
			| 
				316
			 | 
			
				
			 | 
			
			
				-than four.   Example is a bash script syntax error, which returns 2, or 
			 | 
		
	
		
			
			| 
				317
			 | 
			
				
			 | 
			
			
				-Python exception which returns 1.  Yes, in such cases the information 
			 | 
		
	
		
			
			| 
				318
			 | 
			
				
			 | 
			
			
				-conveyed by the exit status is wrong and you should do everything to 
			 | 
		
	
		
			
			| 
				319
			 | 
			
				
			 | 
			
			
				-avoid it. 
			 | 
		
	
		
			
			| 
				320
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				321
			 | 
			
				
			 | 
			
			
				-Possibilities like "test has passed but then something blew up" exist, 
			 | 
		
	
		
			
			| 
				322
			 | 
			
				
			 | 
			
			
				-but conveying this information is responsibility of the test output. 
			 | 
		
	
		
			
			| 
				323
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				324
			 | 
			
				
			 | 
			
			
				-Following table can be used as a cheat-sheet: 
			 | 
		
	
		
			
			| 
				325
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				326
			 | 
			
				
			 | 
			
			
				-    .---------------------------------------------------------------. 
			 | 
		
	
		
			
			| 
				327
			 | 
			
				
			 | 
			
			
				-    | e |    state of         |                                     | 
			 | 
		
	
		
			
			| 
				328
			 | 
			
				
			 | 
			
			
				-    | s |---------------------| script says                         | 
			 | 
		
	
		
			
			| 
				329
			 | 
			
				
			 | 
			
			
				-    |   | SUT   | environment |                                     | 
			 | 
		
	
		
			
			| 
				330
			 | 
			
				
			 | 
			
			
				-    |---|-------|-------------|-------------------------------------| 
			 | 
		
	
		
			
			| 
				331
			 | 
			
				
			 | 
			
			
				-    | 0 | OK    | safe        | test passed, everything worked fine | 
			 | 
		
	
		
			
			| 
				332
			 | 
			
				
			 | 
			
			
				-    | 1 | buggy | safe        | test failed, everything worked fine | 
			 | 
		
	
		
			
			| 
				333
			 | 
			
				
			 | 
			
			
				-    | 2 | ???   | safe        | I decided not to run the test       | 
			 | 
		
	
		
			
			| 
				334
			 | 
			
				
			 | 
			
			
				-    | 3 | ???   | safe        | Something blew up but I managed to  | 
			 | 
		
	
		
			
			| 
				335
			 | 
			
				
			 | 
			
			
				-    |   |       |             | clean up (I promise!)               | 
			 | 
		
	
		
			
			| 
				336
			 | 
			
				
			 | 
			
			
				-    | 4 | ???   | broken      | Something blew up and I rushed out  | 
			 | 
		
	
		
			
			| 
				337
			 | 
			
				
			 | 
			
			
				-    |   |       |             | in panic                            | 
			 | 
		
	
		
			
			| 
				338
			 | 
			
				
			 | 
			
			
				-    | * | ???   | broken      | ...nothing (is dead)                | 
			 | 
		
	
		
			
			| 
				339
			 | 
			
				
			 | 
			
			
				-    '---------------------------------------------------------------' 
			 | 
		
	
		
			
			| 
				340
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				341
			 | 
			
				
			 | 
			
			
				-As you can see, following this semantic allows us to see both the state 
			 | 
		
	
		
			
			| 
				342
			 | 
			
				
			 | 
			
			
				-of the system under test (SUT) *and* the environment. 
			 | 
		
	
		
			
			| 
				343
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				344
			 | 
			
				
			 | 
			
			
				-Following table illustrates how different statuses map to different 
			 | 
		
	
		
			
			| 
				345
			 | 
			
				
			 | 
			
			
				-scenarios with regard to test result as well as state of the environment: 
			 | 
		
	
		
			
			| 
				346
			 | 
			
				
			 | 
			
			
				- 
			 | 
		
	
		
			
			| 
				347
			 | 
			
				
			 | 
			
			
				-    .--------------------------------------------------. 
			 | 
		
	
		
			
			| 
				348
			 | 
			
				
			 | 
			
			
				-    | environment |  test result   |  test result      | 
			 | 
		
	
		
			
			| 
				349
			 | 
			
				
			 | 
			
			
				-    |             | pass fail unkn | pass fail unkn    | 
			 | 
		
	
		
			
			| 
				350
			 | 
			
				
			 | 
			
			
				-    |-------------|----------------|-------------------| 
			 | 
		
	
		
			
			| 
				351
			 | 
			
				
			 | 
			
			
				-    | clean(ed)   |  0    1    3   |  OK  FAIL ERROR   | 
			 | 
		
	
		
			
			| 
				352
			 | 
			
				
			 | 
			
			
				-    | untouched   |  ~    ~    2   |  ~    ~   BAILOUT | 
			 | 
		
	
		
			
			| 
				353
			 | 
			
				
			 | 
			
			
				-    | mess        |  ~    ~    4   |  ~    ~   PANIC   | 
			 | 
		
	
		
			
			| 
				354
			 | 
			
				
			 | 
			
			
				-    | ?! (trap)   |  ~    ~    5   |  ~    ~   ~       | 
			 | 
		
	
		
			
			| 
				355
			 | 
			
				
			 | 
			
			
				-    | ?! (sig 9)  |  ~    ~    137 |  ~    ~   ~       | 
			 | 
		
	
		
			
			| 
				356
			 | 
			
				
			 | 
			
			
				-    | ?! (aliens) |  ~    ~    ?   |  ~    ~   ~       | 
			 | 
		
	
		
			
			| 
				357
			 | 
			
				
			 | 
			
			
				-    '-------------|----------------|-------------------| 
			 | 
		
	
		
			
			| 
				358
			 | 
			
				
			 | 
			
			
				-                  |  exit status   |  human-readable   | 
			 | 
		
	
		
			
			| 
				359
			 | 
			
				
			 | 
			
			
				-                  |                |  name (TF_ES_*)   | 
			 | 
		
	
		
			
			| 
				360
			 | 
			
				
			 | 
			
			
				-                  '------------------------------------' 
			 |