TFKit ===== Installation ------------ The easiest way is to embed TFKit within your repo, ie. clone TFKit and install it using: make install DESTDIR=/path/to/your/repo Now you can run your test suite using *runtests* binary: $ cd /path/to/your/repo $ utils/tfkit/runtest Note that the above probably won't return any useful results as you still don't have any tests. Writing tests ------------- Tests can be written in any scripting language, although the built-in framework, written in Bash, provides some useful features for writing certain kind of relatively simple tests. The harness, though, assumes that: * Any direct sub-directory of `$TF_SUITE` directory ("tests" by default) that contains at least *TF_RUN* executable becomes a test, * basename of this directory becomes the name of the test, * and return code from running the executable is reported as result of the test, according to "Exit status" section below. Naming ------ Test name should start with name of the module that is tested and underscore. If module name contains dots, they should be replaced with underscores as well. core_sanity mod_submod_function ini_iniread are valid test names. Data ---- Should the test need any data, just leave it around in the test directory along with *TF_RUN*. Note that before running, the whole test directory is automatically copied to a temporary location (one per test), and should the test fail, copied back as a debugging relic. For this reason, *do not store huge amounts of data here*. If you really need huge data, consider obtaining it (and throwing it away) within runtime of *TF_RUN*. Exit status ----------- We try hard to follow this semantic: * *Zero* means *OK* -- test has been run and passed. * *One* means *Failure* -- test has been run but failed (e.g. found a bug). * *Two* means *Bailout* -- test has decided not to run at all. * *Three* means *Error* -- there was error detected during execution, but script was able to clean up properly. * *Four* means *Panic* -- there was other error but script *was not* able to clean up properly. * Anything else should indicate other uncaught errors, including those outside control of the program such as segfaults in the test code or test being SIGKILLed. Notice that the higher the value is, the worse situation it indicates. Thus, if a test is composed of several sub-tests, you need to make sure to always **exit with the highest value** (subtest.sh does take care of this). See *common.sh* for functions and variables to help with handling exit statuses with this semantic. Also see Notes section for more details on exit statuses, including cheat sheet and dscussuion. Framework --------- ### harness.sh ### This part is not intended to be used in tests, but rather contains functions that help govern test discovery, preparation and execution as is described in previous sections. Feel free to poke around, of course. ### subtest.sh ### As name suggests, this file defines few functions to handle subtests in *TF_RUN*. In order to make use of the subtests functionality, you will need to define two functions yourself: `tf_enum_subtests` to enumerate names of tests you want to run, and `tf_do_subtest` with actual test implementation. The minimal *TF_RUN* with two subtests could look like this: #!/bin/bash . $TF_DIR/include/subtest.sh tf_enum_subtests() { echo test1 echo test2 something && echo test3 } tf_do_subtest() { case $1 in test1) myprog foo ;; test2) myprog bar ;; test3) myprog baz ;; esac } tf_do_subtests At the end, `tf_do_subtests` acts as a launcher of the actual test. In short, it will 1. run `tf_enum_subtests`, taking each line as name of a subtest; for each subtest: 1. source *TF_SETUP*, if such file is found, 2. launch the `tf_do_subtest()` function with subtest name as the only argument, 3. source *TF_CLEANUP*, if such file is found, 2. and finally, report "worst" exit status encountered. Note that subtest names need to be single words (`[a-zA-Z0-9_]`). ### tools.sh ### This file contains various tools and utilities to help with testing. Curently there is only one function, `tf_testflt` designed to help write tests for simple unix filters. #### tf_testflt #### The idea is that tester specifies * test name, * command to launch the system under test, * a data stream to use as STDIN, * and expected STDOUT, STDERR, and exit status. and tf_testflt launches the command, collects tha data and evaluates and reports the result using unified diff. In its simplest form: tf_testflt -n foo my_command arg the function will run `my_command arg` (not piping anything to it), and will expect it to finish with exit status 0 and empty both STDERR and STDOUT. Example of full form, tf_testflt -n foo -i foo.in -O foo.stdout -E foo.stderr -S 2 myprog will pipe foo.in into `myprog`, expecting exit status of 2, and STDOUT and STDERR as above. Notice that parameters specifying expected values are uppercase, and those specifying input values are lowercase. Specifying name is mandatory, because it's used in reporting messages, and as a basis for naming temporary result files: these are saved in *results* subdirectory and kept for further reference. ### common.sh ### This includes simple functions and variables shared between both mentioned libraries. First group is designed to help support the exit status semantic: * The functions are `tf_exit_pass`, `tf_exit_fail`, `tf_exit_bailout`, `tf_exit_error` and `tf_exit_panic` and each take any number of parameters that are printed on stderr. * The variables are `TF_ES_OK`, `TF_ES_FAIL`, `TF_ES_BAILOUT`, `TF_ES_ERROR` and `TF_ES_PANIC` and are supposed to be used with `return` builtin, e.g. to return from `tf_exit_error`. Second group is useful to better control output: functions `tf_warn`, `tf_debug` and `tf_think` are used to print stuff on STDERR. Use of `tf_warn` is apparent, just as `tf_debug`, the latter being muted if `TF_DEBUG` is set to `false` (set it to `true` to turn on debugging). `tf_think` is used for progress info, and is muted unless `TF_VERBOSE` is set to `true`, which is by default. ### Setup and cleanup ### Special files *TF_SETUP* and *TF_CLEANUP* (one of them or both) can be added along with *TF_RUN*. These will be sourced before (*TF_SETUP*) and after every subtest (*TF_CLEANUP*). First, if any of these files are missing, it is considered as if the respective phase succeeded. Second, if setup phase fails, test will be skipped and subtest exit status will be *TF_ES_BAILOUT*. Last, if cleanup fails (no matter result of setup), subtests aborts with *TF_ES_PANIC* returned. Be aware that in this case the actual test status, albeit useful, is lost. When coming from other test frameworks, this may feel harsh, but note that this has been designed with the idea that if a cleanup fails, it may render all further tests are automatically unsafe, because the environment is not as expected. To cope with this behavior, try to bear in mind following advice: 1. Make sure you write setup/cleanup procedures with extreme care and test them well. 2. Do not do complicated and risky things in the setup/cleanup phases. 3. If you need to do such things, consider doing them in the *TF_RUN* instead of doing them for all subtests. 4. You don't need to clean up everything, the contents of the testing dir will be moved out from the test system. 5. If there are scenarios you can safely fix or ignore, handle them in a robust manner. Notes ----- ### bailout vs. `tf_enum_subtests` ### One more note to claify relation of bailout and `tf_enum_subtests`. As you may have noticed, there are two ways how to skip a test: return prematurely with `TF_ES_BAILOUT`, or suppress enumeration in `tf_enum_subtests`. The problem is that the latter does not do anything to inform upper in the stack that a test has been skipped, which seems to break the principle described in previous sections. Don't confuse these mechanisms, though. Each is supposed to be used for distinct purpose. Compare: by using the `tf_enum_subtests` you are saying that you actually **did not even want** to run the test in the first place. By using `TF_ES_BAILOUT`, you are saying that you **wanted** to run the test but could not. A few common cases if that helps you: * If during the test you find out that for some reason it can't be carried out (e.g. an external resource is not available, or something outside the SUT is broken), use `TF_ES_BAILOUT`. tf_enum_subtests() { echo test1 echo test2 echo test3 } tf_do_subtest() { case $1 in test1) do_stuff ;; test2) do_other_stuff ;; test3) curl -s http://www.example.com/ >file \ || return $TF_ES_BAILOUT do_stuff_with file ;; esac } * If you want to filter out some sub-tests for some platforms, e.g. a test for only 64-bit architectures, or a test only for Mac OS (IOW, you can safely say that running this sub-test would be totally pointless on this box), use `tf_enum_subtests`--just omit this test from enumeration. tf_enum_subtests() { echo test1 echo test2 if this_is_macos_x; then echo test3 fi } * If you want to disable (comment out test) that you might not have implemented yet or is broken (and for some reason you still want it to haunt the test code) or something else outside SUT is broken and prevents you from running the test, use `tf_enum_subtests` and properly comment the reasons in code. tf_enum_subtests() { echo test1 echo test2 # echo test3 #FIXME: implement after bz1234 } * If in doubt, use `TF_ES_BAILOUT`. ### On exit statuses: three and above ### The difference in *error*, *panic* and higher values is subtle but important. Follow me as I try to explain: 1. If script has changed something on the system outside the working directory, it is apparently expected to revert that change. 2. Now if an error occurs, but the code responsible for cleaning up is safely run, you can say there was *error but we have recovered*. 3. But if the change can't be reverted safely, we know that we have broken something and latter code may lead to weird results (including masking bugs(!)), it's time to *panic* (in the code, not in real life ;)) 4. And then there are corner cases like a bug in the script, OOM kill or timeout when the status will be different and not really controlled by the script. Such cases will have to be treated the same way as the "panic" case, but... 5. the use of *panic* adds hint that the status has been set consciously by the script, albeit exiting "in a hurry"--without proper clean up. Unfortunately there will be cases like above but with the error code less than four. Example is a Bash script syntax error, which returns 2, or Python exception which returns 1. Yes, in such cases the information conveyed by the exit status is wrong and you should do everything to avoid it. Possibilities like "test has passed but then something blew up" exist, but conveying this information is responsibility of the test output. Following table can be used as a cheat-sheet: .---------------------------------------------------------------. | e | state of | | | s |---------------------| script says | | | SUT | environment | | |---|-------|-------------|-------------------------------------| | 0 | OK | safe | test passed, everything worked fine | | 1 | buggy | safe | test failed, everything worked fine | | 2 | ??? | safe | I decided not to run the test | | 3 | ??? | safe | Something blew up but I managed to | | | | | clean up (I promise!) | | 4 | ??? | broken | Something blew up and I rushed out | | | | | in panic | | * | ??? | broken | ...nothing (is dead) | '---------------------------------------------------------------' As you can see, following this semantic allows us to see both the state of the system under test (SUT) *and* the environment. Following table illustrates how different statuses map to different scenarios with regard to test result as well as state of the environment: .--------------------------------------------------. | environment | test result | test result | | | pass fail unkn | pass fail unkn | |-------------|----------------|-------------------| | clean(ed) | 0 1 3 | OK FAIL ERROR | | untouched | ~ ~ 2 | ~ ~ BAILOUT | | mess | ~ ~ 4 | ~ ~ PANIC | | ?! (trap) | ~ ~ 5 | ~ ~ ~ | | ?! (sig 9) | ~ ~ 137 | ~ ~ ~ | | ?! (aliens) | ~ ~ ? | ~ ~ ~ | '-------------|----------------|-------------------| | exit status | human-readable | | | name (TF_ES_*) | '------------------------------------'