shellfu/test

Tests

Running tests is handled by run_tests in setup/mk.sh.

Tests can be written in any scripting language--they just need to be put into tests/ directory, equipped with proper shebang line and marked as executable.

Naming

Test filename should start with name of the module that is tested and underscore. If module name contains dots, they should be replaced with underscores as well.

core_sanity
mod_submod_function
ini_iniread

are valid test names.

Data

If test needs data, place it under test/data/NAME, where NAME is exact filename of the test. The data will then be available in the working directory where the test is started (a temporary directory).

Exit status

We try hard to follow this semantic:

• Zero means that test has been run and passed

• One means that test has been run but failed.

• Two means that test has bailed out.

• Three means that there was error detected during execution.

• Four or anything else means that there were other errors that script was not able to detect.

Notice that the higher the value is, the worse situation it signifies. Thus, if a test is composed of several sub-tests, always exit with the highest value. The same applies for the run_tests routine itself.

Three and above

The subtle but important difference in three, four and "anything else" is, that if script has changed something on the system outside the working directory, it is apparently expected to revert that change. Now if an error occurs, but the code responsible for cleaning up is safely run, you can use three. But if the change can't be reverted safely, use four.

Apparently there may be corner cases like a bug in the script, OOM kill or timeout when the status will be different and not really controlled by the script. There is underlying assumption, though, that such exit statuses are higher than four. Therefore, while from perspective of environment safety, four has the same meaning as anything higher ("something blew up and system is broken"), however, the use of four adds hint that the status has been set consciously by the script, albeit exiting "in a hurry"--without proper clean up.

Unfortunately there will be cases like above but with the error code less than four. Example is a bash script syntax error, which returns 2, or Python exception which returns 1. Yes, in such cases the information conveyed by the exit status is wrong and you should do everything to avoid it.

Possibilities like "test has passed but then something blew up" exist, but conveying this information is repoomnsibility of the test output.

Following table can be used as a cheat-sheet:

.---------------------------------------------------------------.
| e |    state of         |                                     |
| s |---------------------| script says                         |
|   | SUT   | environment |                                     |
|---|-------|-------------|-------------------------------------|
| 0 | OK    | safe        | test passed, everything worked fine |
| 1 | buggy | safe        | test failed, everything worked fine |
| 2 | ???   | safe        | I decided not to run the test       |
| 3 | ???   | broken      | Something blew up but I managed to  |
|   |       |             | clean up (I promise!)               |
| 4 | ???   | broken      | Something blew up and I rushed out  |
|   |       |             | in panic                            |
| * | ???   | broken      | ...nothing (is dead)                |
'---------------------------------------------------------------'

As you can see, following this semantic allows us to see both the state of the system umder test (SUT) and the environment.