Remove obsolete files (moved to doc/ long ago)

utils/tfkit/README.md

@@ -1,15 +0,0 @@
-TFKIT
-=====
-
-A simple portable (or rather "movable") test framework.
-
- 1. Copy (or `git submodule`) *test* sub-folder to your place
-    where you want to run tests.
-
- 2. Run your framework by `./tfkit/runtests`.
-
- 3. Don't be surprised--you haven't added any tests yet :)
-
- 4. Read tfkit/README.testing.md
-
- 5. Start adding tests to `tests` folder

utils/tfkit/README.testing.md

@@ -1,360 +0,0 @@
-Tests
-=====
-
-Running tests is handled by tfkit/runtests:
-
-    $ tfkit/runtest [filter]
-
-*filter* is a regular expression to be applied to sub-test name, running
-only the matching ones.  See below for details.
-
-
-Writing tests
--------------
-
-Tests can be written in any scripting language, although the built-in
-framework, written in Bash, provides some useful features for writing
-certain kind of relatively simple tests.
-
-The harness, though, assumes that:
-
- *  Any direct sub-directory of `$TF_SUITE` directory ("tests" by default)
-    that contains at least *TF_RUN* executable becomes a test,
-
- *  basename of this directory becomes the name of the test,
-
- *  and return code from running the executable is reported
-    as result of the test, according to "Exit status" chapter below.
-
-
-Naming
-------
-
-Test name should start with name of the module that is tested and
-underscore.  If module name contains dots, they should be replaced with
-underscores as well.
-
-    core_sanity
-    mod_submod_function
-    ini_iniread
-
-are valid test names.
-
-
-Data
-----
-
-Should the test need any data, just leave it around in the test directory
-along with *TF_RUN*.
-
-Note that before running, the whole test directory is automatically
-copied to a temporary location (one per test), and should the test fail,
-copied back as a debugging artifact.  For this reason, *do not store
-huge amounts of data here*.  If you really need huge data, consider
-obtaining it (and throwing it away) within runtime of *TF_RUN*.
-
-
-Exit status
------------
-
-We try hard to follow this semantic:
-
- *  *Zero* means *OK* -- test has been run and passed.
-
- *  *One* means *Failure* -- test has been run but failed (e.g. found
-     a bug).
-
- *  *Two* means *Bailout* --  test has decided not to run at all.
-
- *  *Three* means *Error* -- there was error detected during execution,
-     but script was able to clean up properly.
-
- *  *Four* means *Panic* -- there was other error but script *was not*
-     able to clean up properly.
-
- *  Anything else should indicate other uncaught errors, including those
-    outside control of the program such as segfaults in the test code
-    or test being SIGKILLed.
-
-Notice that the higher the value is, the worse situation it indicates.
-Thus, if a test is composed of several sub-tests, you need to make sure
-to always **exit with the highest value** (subtest.sh does take care
-of this).
-
-See *common.sh* for functions and variables to help with handling exit
-statuses with this semantic.
-
-Also see Notes section for more details on exit statuses, including
-cheat sheet and dscussuion.
-
-
-Framework
----------
-
-
-### harness.sh ###
-
-This part is not intended to be used in tests, but rather contains
-functions that help govern test discovery, preparation and execution as
-is described in previous chapters.  Feel free to poke around, of course.
-
-
-### subtest.sh ###
-
-As name suggests, this file defines few functions to handle subtests
-in *TF_RUN*.
-
-In order to make use of the subtests functionality, you will need to
-define two functions yourself:  `tf_enum_subtests` to enumerate names of
-tests you want to run, and `tf_name2cmd` to translate each name an actual
-command that would perform it and return with the the correct exit status.
-
-The minimal *TF_RUN* with two subtests could look like this:
-
-    #!/bin/bash
-
-    . $TF_DIR/include/subtest.sh
-
-    tf_enum_subtests() {
-        echo test1
-        echo test2
-        something && echo test3
-    }
-
-    tf_name2cmd() {
-        case $1 in
-            test1)  echo myprog foo ;;
-            test2)  echo myprog bar ;;
-        esac
-    }
-
-    tf_do_subtests
-
-At the end, `tf_do_subtests` acts as a launcher of the actual test.
-In short, it will
-
- *  take each enumerated subtest from `tf_enum_subtests`,
- *  source *TF_SETUP*, if such file is found,
- *  translate te subtest name to a command,
- *  launch the command,
- *  source *TF_CLEANUP*, if such file is found,
- *  and report "worst" exit status encountered.
-
-All but the first and last step is done by `tf_do_subtest`, so in some
-cases you may want to re-define this one as well.
-
-Note that subtest names need to be single words (`[a-zA-Z0-9_]`).
-
-
-### tools.sh ###
-
-This file contains various tools and utilities to help with testing.
-
-Curently there is only one function, `tf_testflt` designed to help write
-tests for simple unix filters.
-
-
-#### tf_testflt ####
-
-The idea is that tester specifies
-
- *  test name,
- *  command to launch the system under test,
- *  a data stream to use as STDIN,
- *  and expected STDOUT, STDERR, and exit status.
-
-and tf_testflt launches the command, collects tha data and evaluates
-and reports the result using unified diff.
-
-In its simplest form:
-
-    tf_testflt -n foo my_command arg
-
-the function will run `my_command arg` (not piping anything to it),
-and will expect it to finish with exit status 0 and empty both STDERR
-and STDOUT.
-
-Example of full form,
-
-    tf_testflt -n foo -i foo.in -O foo.stdout -E foo.stderr -S 2 myprog
-
-will pipe foo.in into `myprog`, expecting exit status of 2, and STDOUT and
-STDERR as above.  Notice that parameters specifying expected values are
-uppercase, and those specifying input values are lowercase.
-
-Specifying name is mandatory, because it's used in reporting messages,
-and as a basis for naming temporary result files: these are saved in
-*results* subdirectory and kept for further reference.
-
-
-### common.sh ###
-
-This includes simple functions and variables shared between both mentioned
-libraries.
-
-First group is designed to help support the exit status semantic:
-
- *  The functions are `tf_exit_pass`, `tf_exit_fail`, `tf_exit_bailout`,
-    `tf_exit_error` and `tf_exit_panic` and each take any number of
-    parameters that are printed on stderr.
-
- *  The variables are `TF_ES_OK`, `TF_ES_FAIL`, `TF_ES_BAILOUT`,
-    `TF_ES_ERROR` and `TF_ES_PANIC` and are supposed to be used with
-    `return` builtin, e.g. to return from `tf_exit_error`.
-
-Second group is useful to better control output:  functions `tf_warn`,
-`tf_debug` and `tf_think` are used to print stuff on STDERR.  Use of
-`tf_warn` is apparent, just as `tf_debug`, the latter being muted if
-`TF_DEBUG` is set to `false` (set it to `true` to turn on debugging).
-
-`tf_think` is used for progress info, and is muted unless `TF_VERBOSE`
-is set to `true`, which is by default.
-
-
-### Setup and cleanup ###
-
-Special files *TF_SETUP* and *TF_CLEANUP* (one of them or both) can be
-added along with *TF_RUN*.  These will be sourced before (*TF_SETUP*)
-and after every subtest (*TF_CLEANUP*).
-
-First, if any of these files are missing, it is considered as if the
-respective phase succeeded.  Second, if setup phase fails, test will
-be skipped and subtest exit status will be *TF_ES_BAILOUT*.   Last,
-if cleanup fails (no matter result of setup), subtests aborts with
-*TF_ES_PANIC* returned.  Be aware that in this case the actual test
-status, albeit useful, is lost.
-
-When coming from other test frameworks, this may feel harsh, but note
-that this has been designed with the idea that if a cleanup fails,
-it may render all further tests are automatically unsafe, because the
-environment is not as expected.
-
-To cope with this behavior, try to bear in mind following advice:
-
- 1. Make sure you write setup/cleanup procedures with extreme care and
-    test them well.
-
- 2. Do not do complicated and risky things in the setup/cleanup phases.
-
- 3. If you need to do such things, consider doing them in the *TF_RUN*
-    instead of doing them for all subtests.
-
- 4. You don't need to clean up everything, the contents of the testing dir
-    will be moved out from the test system.
-
- 5. If there are scenarios you can safely fix or ignore, handle them in
-    a robust manner.
-
-
-Notes
------
-
-
-### bailout vs. `tf_enum_subtests` ###
-
-One more note to claify relation of bailout and `tf_enum_subtests`.
-As you may have noticed, there are two ways how to skip a test:
-return prematurely with `TF_ES_BAILOUT`, or suppress enumeration in
-`tf_enum_subtests`.  The problem is that the latter does not do anything
-to inform upper in the stack that a test has been skipped, which seems to
-break the principle described in the previous chapters.
-
-Don't confuse these mechanisms, though. Each is supposed to be used
-for distinct purpose.  Compare: by using the `tf_enum_subtests` you are
-saying that you actually **did not even want** to run the test in the
-first place.  By using `TF_ES_BAILOUT`, you are saying that you **wanted**
-to run the test but could not.
-
-A few common cases if that helps you:
-
- *  If during the test you find out that for some reason it can't be
-    carried out (e.g. an external resource is not available, or
-    something outside the SUT is broken), use `TF_ES_BAILOUT`.
-
- *  If you want to disable the test because for some long-term condition,
-    e.g. a known bug outside SUT but preventing execution of the test
-    is not fixed, use `tf_enum_subtests`.
-
- *  If you want to filter out some sub-tests to only for some platforms,
-    e.g. 64-bit architecture, (IOW, you can safely check that a
-    sub-test would be totally pointless if run on this box), use
-    `tf_enum_subtests`.
-
- *  If you want to disable (comment out test) that you might not have
-    implemented yet or is broken (and for some reason you still want
-    it to haunt the test code), use `tf_enum_subtests` and properly
-    comment the reasons in code.
-
- *  If in doubt, use `TF_ES_BAILOUT`.
-
-
-### On exit statuses: three and above ###
-
-The difference in *error*, *panic* and higher values is subtle but
-important.  Follow me as I try to explain:
-
- 1. If script has changed something on the system outside the working
-    directory, it is apparently expected to revert that change.
-
- 2. Now if an error occurs, but the code responsible for cleaning up is
-    safely run, you can say there was *error but we have recovered*.
-
- 3. But if the change can't be reverted safely, we know that we have
-    broken something and latter code may lead to weird results (including
-    masking bugs(!)), it's time to *panic* (in the code, not in real
-    life ;))
-
- 4. And then there are corner cases like a bug in the script, OOM kill
-    or timeout when the status will be different and not really controlled
-    by the script.  Such cases will have to be treated the same way as
-    the "panic" case, but...
-
- 5. the use of *panic* adds hint that the status has been set consciously
-    by the script, albeit exiting "in a hurry"--without proper clean up.
-
-Unfortunately there will be cases like above but with the error code less
-than four.   Example is a bash script syntax error, which returns 2, or
-Python exception which returns 1.  Yes, in such cases the information
-conveyed by the exit status is wrong and you should do everything to
-avoid it.
-
-Possibilities like "test has passed but then something blew up" exist,
-but conveying this information is responsibility of the test output.
-
-Following table can be used as a cheat-sheet:
-
-    .---------------------------------------------------------------.
-    | e |    state of         |                                     |
-    | s |---------------------| script says                         |
-    |   | SUT   | environment |                                     |
-    |---|-------|-------------|-------------------------------------|
-    | 0 | OK    | safe        | test passed, everything worked fine |
-    | 1 | buggy | safe        | test failed, everything worked fine |
-    | 2 | ???   | safe        | I decided not to run the test       |
-    | 3 | ???   | safe        | Something blew up but I managed to  |
-    |   |       |             | clean up (I promise!)               |
-    | 4 | ???   | broken      | Something blew up and I rushed out  |
-    |   |       |             | in panic                            |
-    | * | ???   | broken      | ...nothing (is dead)                |
-    '---------------------------------------------------------------'
-
-As you can see, following this semantic allows us to see both the state
-of the system under test (SUT) *and* the environment.
-
-Following table illustrates how different statuses map to different
-scenarios with regard to test result as well as state of the environment:
-
-    .--------------------------------------------------.
-    | environment |  test result   |  test result      |
-    |             | pass fail unkn | pass fail unkn    |
-    |-------------|----------------|-------------------|
-    | clean(ed)   |  0    1    3   |  OK  FAIL ERROR   |
-    | untouched   |  ~    ~    2   |  ~    ~   BAILOUT |
-    | mess        |  ~    ~    4   |  ~    ~   PANIC   |
-    | ?! (trap)   |  ~    ~    5   |  ~    ~   ~       |
-    | ?! (sig 9)  |  ~    ~    137 |  ~    ~   ~       |
-    | ?! (aliens) |  ~    ~    ?   |  ~    ~   ~       |
-    '-------------|----------------|-------------------|
-                  |  exit status   |  human-readable   |
-                  |                |  name (TF_ES_*)   |
-                  '------------------------------------'