Update TFKit to v0.0.2

utils/tfkit/doc/README.md

@@ -0,0 +1,360 @@
+Tests
+=====
+
+Running tests is handled by tfkit/runtests:
+
+    $ tfkit/runtest [filter]
+
+*filter* is a regular expression to be applied to sub-test name, running
+only the matching ones.  See below for details.
+
+
+Writing tests
+-------------
+
+Tests can be written in any scripting language, although the built-in
+framework, written in Bash, provides some useful features for writing
+certain kind of relatively simple tests.
+
+The harness, though, assumes that:
+
+ *  Any direct sub-directory of `$TF_SUITE` directory ("tests" by default)
+    that contains at least *TF_RUN* executable becomes a test,
+
+ *  basename of this directory becomes the name of the test,
+
+ *  and return code from running the executable is reported
+    as result of the test, according to "Exit status" chapter below.
+
+
+Naming
+------
+
+Test name should start with name of the module that is tested and
+underscore.  If module name contains dots, they should be replaced with
+underscores as well.
+
+    core_sanity
+    mod_submod_function
+    ini_iniread
+
+are valid test names.
+
+
+Data
+----
+
+Should the test need any data, just leave it around in the test directory
+along with *TF_RUN*.
+
+Note that before running, the whole test directory is automatically
+copied to a temporary location (one per test), and should the test fail,
+copied back as a debugging artifact.  For this reason, *do not store
+huge amounts of data here*.  If you really need huge data, consider
+obtaining it (and throwing it away) within runtime of *TF_RUN*.
+
+
+Exit status
+-----------
+
+We try hard to follow this semantic:
+
+ *  *Zero* means *OK* -- test has been run and passed.
+
+ *  *One* means *Failure* -- test has been run but failed (e.g. found
+     a bug).
+
+ *  *Two* means *Bailout* --  test has decided not to run at all.
+
+ *  *Three* means *Error* -- there was error detected during execution,
+     but script was able to clean up properly.
+
+ *  *Four* means *Panic* -- there was other error but script *was not*
+     able to clean up properly.
+
+ *  Anything else should indicate other uncaught errors, including those
+    outside control of the program such as segfaults in the test code
+    or test being SIGKILLed.
+
+Notice that the higher the value is, the worse situation it indicates.
+Thus, if a test is composed of several sub-tests, you need to make sure
+to always **exit with the highest value** (subtest.sh does take care
+of this).
+
+See *common.sh* for functions and variables to help with handling exit
+statuses with this semantic.
+
+Also see Notes section for more details on exit statuses, including
+cheat sheet and dscussuion.
+
+
+Framework
+---------
+
+
+### harness.sh ###
+
+This part is not intended to be used in tests, but rather contains
+functions that help govern test discovery, preparation and execution as
+is described in previous chapters.  Feel free to poke around, of course.
+
+
+### subtest.sh ###
+
+As name suggests, this file defines few functions to handle subtests
+in *TF_RUN*.
+
+In order to make use of the subtests functionality, you will need to
+define two functions yourself:  `tf_enum_subtests` to enumerate names of
+tests you want to run, and `tf_name2cmd` to translate each name an actual
+command that would perform it and return with the the correct exit status.
+
+The minimal *TF_RUN* with two subtests could look like this:
+
+    #!/bin/bash
+
+    . $TF_DIR/include/subtest.sh
+
+    tf_enum_subtests() {
+        echo test1
+        echo test2
+        something && echo test3
+    }
+
+    tf_name2cmd() {
+        case $1 in
+            test1)  echo myprog foo ;;
+            test2)  echo myprog bar ;;
+        esac
+    }
+
+    tf_do_subtests
+
+At the end, `tf_do_subtests` acts as a launcher of the actual test.
+In short, it will
+
+ *  take each enumerated subtest from `tf_enum_subtests`,
+ *  source *TF_SETUP*, if such file is found,
+ *  translate te subtest name to a command,
+ *  launch the command,
+ *  source *TF_CLEANUP*, if such file is found,
+ *  and report "worst" exit status encountered.
+
+All but the first and last step is done by `tf_do_subtest`, so in some
+cases you may want to re-define this one as well.
+
+Note that subtest names need to be single words (`[a-zA-Z0-9_]`).
+
+
+### tools.sh ###
+
+This file contains various tools and utilities to help with testing.
+
+Curently there is only one function, `tf_testflt` designed to help write
+tests for simple unix filters.
+
+
+#### tf_testflt ####
+
+The idea is that tester specifies
+
+ *  test name,
+ *  command to launch the system under test,
+ *  a data stream to use as STDIN,
+ *  and expected STDOUT, STDERR, and exit status.
+
+and tf_testflt launches the command, collects tha data and evaluates
+and reports the result using unified diff.
+
+In its simplest form:
+
+    tf_testflt -n foo my_command arg
+
+the function will run `my_command arg` (not piping anything to it),
+and will expect it to finish with exit status 0 and empty both STDERR
+and STDOUT.
+
+Example of full form,
+
+    tf_testflt -n foo -i foo.in -O foo.stdout -E foo.stderr -S 2 myprog
+
+will pipe foo.in into `myprog`, expecting exit status of 2, and STDOUT and
+STDERR as above.  Notice that parameters specifying expected values are
+uppercase, and those specifying input values are lowercase.
+
+Specifying name is mandatory, because it's used in reporting messages,
+and as a basis for naming temporary result files: these are saved in
+*results* subdirectory and kept for further reference.
+
+
+### common.sh ###
+
+This includes simple functions and variables shared between both mentioned
+libraries.
+
+First group is designed to help support the exit status semantic:
+
+ *  The functions are `tf_exit_pass`, `tf_exit_fail`, `tf_exit_bailout`,
+    `tf_exit_error` and `tf_exit_panic` and each take any number of
+    parameters that are printed on stderr.
+
+ *  The variables are `TF_ES_OK`, `TF_ES_FAIL`, `TF_ES_BAILOUT`,
+    `TF_ES_ERROR` and `TF_ES_PANIC` and are supposed to be used with
+    `return` builtin, e.g. to return from `tf_exit_error`.
+
+Second group is useful to better control output:  functions `tf_warn`,
+`tf_debug` and `tf_think` are used to print stuff on STDERR.  Use of
+`tf_warn` is apparent, just as `tf_debug`, the latter being muted if
+`TF_DEBUG` is set to `false` (set it to `true` to turn on debugging).
+
+`tf_think` is used for progress info, and is muted unless `TF_VERBOSE`
+is set to `true`, which is by default.
+
+
+### Setup and cleanup ###
+
+Special files *TF_SETUP* and *TF_CLEANUP* (one of them or both) can be
+added along with *TF_RUN*.  These will be sourced before (*TF_SETUP*)
+and after every subtest (*TF_CLEANUP*).
+
+First, if any of these files are missing, it is considered as if the
+respective phase succeeded.  Second, if setup phase fails, test will
+be skipped and subtest exit status will be *TF_ES_BAILOUT*.   Last,
+if cleanup fails (no matter result of setup), subtests aborts with
+*TF_ES_PANIC* returned.  Be aware that in this case the actual test
+status, albeit useful, is lost.
+
+When coming from other test frameworks, this may feel harsh, but note
+that this has been designed with the idea that if a cleanup fails,
+it may render all further tests are automatically unsafe, because the
+environment is not as expected.
+
+To cope with this behavior, try to bear in mind following advice:
+
+ 1. Make sure you write setup/cleanup procedures with extreme care and
+    test them well.
+
+ 2. Do not do complicated and risky things in the setup/cleanup phases.
+
+ 3. If you need to do such things, consider doing them in the *TF_RUN*
+    instead of doing them for all subtests.
+
+ 4. You don't need to clean up everything, the contents of the testing dir
+    will be moved out from the test system.
+
+ 5. If there are scenarios you can safely fix or ignore, handle them in
+    a robust manner.
+
+
+Notes
+-----
+
+
+### bailout vs. `tf_enum_subtests` ###
+
+One more note to claify relation of bailout and `tf_enum_subtests`.
+As you may have noticed, there are two ways how to skip a test:
+return prematurely with `TF_ES_BAILOUT`, or suppress enumeration in
+`tf_enum_subtests`.  The problem is that the latter does not do anything
+to inform upper in the stack that a test has been skipped, which seems to
+break the principle described in the previous chapters.
+
+Don't confuse these mechanisms, though. Each is supposed to be used
+for distinct purpose.  Compare: by using the `tf_enum_subtests` you are
+saying that you actually **did not even want** to run the test in the
+first place.  By using `TF_ES_BAILOUT`, you are saying that you **wanted**
+to run the test but could not.
+
+A few common cases if that helps you:
+
+ *  If during the test you find out that for some reason it can't be
+    carried out (e.g. an external resource is not available, or
+    something outside the SUT is broken), use `TF_ES_BAILOUT`.
+
+ *  If you want to disable the test because for some long-term condition,
+    e.g. a known bug outside SUT but preventing execution of the test
+    is not fixed, use `tf_enum_subtests`.
+
+ *  If you want to filter out some sub-tests to only for some platforms,
+    e.g. 64-bit architecture, (IOW, you can safely check that a
+    sub-test would be totally pointless if run on this box), use
+    `tf_enum_subtests`.
+
+ *  If you want to disable (comment out test) that you might not have
+    implemented yet or is broken (and for some reason you still want
+    it to haunt the test code), use `tf_enum_subtests` and properly
+    comment the reasons in code.
+
+ *  If in doubt, use `TF_ES_BAILOUT`.
+
+
+### On exit statuses: three and above ###
+
+The difference in *error*, *panic* and higher values is subtle but
+important.  Follow me as I try to explain:
+
+ 1. If script has changed something on the system outside the working
+    directory, it is apparently expected to revert that change.
+
+ 2. Now if an error occurs, but the code responsible for cleaning up is
+    safely run, you can say there was *error but we have recovered*.
+
+ 3. But if the change can't be reverted safely, we know that we have
+    broken something and latter code may lead to weird results (including
+    masking bugs(!)), it's time to *panic* (in the code, not in real
+    life ;))
+
+ 4. And then there are corner cases like a bug in the script, OOM kill
+    or timeout when the status will be different and not really controlled
+    by the script.  Such cases will have to be treated the same way as
+    the "panic" case, but...
+
+ 5. the use of *panic* adds hint that the status has been set consciously
+    by the script, albeit exiting "in a hurry"--without proper clean up.
+
+Unfortunately there will be cases like above but with the error code less
+than four.   Example is a bash script syntax error, which returns 2, or
+Python exception which returns 1.  Yes, in such cases the information
+conveyed by the exit status is wrong and you should do everything to
+avoid it.
+
+Possibilities like "test has passed but then something blew up" exist,
+but conveying this information is responsibility of the test output.
+
+Following table can be used as a cheat-sheet:
+
+    .---------------------------------------------------------------.
+    | e |    state of         |                                     |
+    | s |---------------------| script says                         |
+    |   | SUT   | environment |                                     |
+    |---|-------|-------------|-------------------------------------|
+    | 0 | OK    | safe        | test passed, everything worked fine |
+    | 1 | buggy | safe        | test failed, everything worked fine |
+    | 2 | ???   | safe        | I decided not to run the test       |
+    | 3 | ???   | safe        | Something blew up but I managed to  |
+    |   |       |             | clean up (I promise!)               |
+    | 4 | ???   | broken      | Something blew up and I rushed out  |
+    |   |       |             | in panic                            |
+    | * | ???   | broken      | ...nothing (is dead)                |
+    '---------------------------------------------------------------'
+
+As you can see, following this semantic allows us to see both the state
+of the system under test (SUT) *and* the environment.
+
+Following table illustrates how different statuses map to different
+scenarios with regard to test result as well as state of the environment:
+
+    .--------------------------------------------------.
+    | environment |  test result   |  test result      |
+    |             | pass fail unkn | pass fail unkn    |
+    |-------------|----------------|-------------------|
+    | clean(ed)   |  0    1    3   |  OK  FAIL ERROR   |
+    | untouched   |  ~    ~    2   |  ~    ~   BAILOUT |
+    | mess        |  ~    ~    4   |  ~    ~   PANIC   |
+    | ?! (trap)   |  ~    ~    5   |  ~    ~   ~       |
+    | ?! (sig 9)  |  ~    ~    137 |  ~    ~   ~       |
+    | ?! (aliens) |  ~    ~    ?   |  ~    ~   ~       |
+    '-------------|----------------|-------------------|
+                  |  exit status   |  human-readable   |
+                  |                |  name (TF_ES_*)   |
+                  '------------------------------------'

utils/tfkit/doc/templates/grep_engine/TF_RUN

@@ -1,7 +1,7 @@
 #!/bin/bash
 
-. $TF_DIR/include/subtest.sh
-. $TF_DIR/include/tools.sh
+. "$TF_DIR/include/subtest.sh"
+. "$TF_DIR/include/tools.sh"
 
 tf_enum_subtests() {
     echo fixed

utils/tfkit/include/common.sh

@@ -102,7 +102,7 @@ tf_debug() {
     for msg in "$@";
     do
         $TF_COLOR && echo -ne "$TF_COLOR_CYAN" >&2
-        echo "||| $1" >&2;
+        echo "||| $msg" >&2;
         $TF_COLOR && echo -ne "$TF_COLOR_NONE" >&2
     done
 }
@@ -116,7 +116,7 @@ tf_think() {
     for msg in "$@";
     do
         $TF_COLOR && echo -ne "$TF_COLOR_LBLACK" >&2
-        echo "$pfx$1$sfx" >&2;
+        echo "$pfx$msg$sfx" >&2;
         $TF_COLOR && echo -ne "$TF_COLOR_NONE" >&2
     done
 }
@@ -129,7 +129,7 @@ tf_warn() {
     for msg in "$@";
     do
         $TF_COLOR && echo -ne "$TF_COLOR_LRED" >&2
-        echo "$1" >&2;
+        echo "$msg" >&2;
         $TF_COLOR && echo -ne "$TF_COLOR_NONE" >&2
     done
 }

utils/tfkit/include/harness.sh

@@ -2,7 +2,7 @@
 # ffoo test harness
 # See LICENSE file for copyright and license details.
 
-. $TF_DIR/include/common.sh
+. "$TF_DIR/include/common.sh"
 
 #
 # Default path to header generator
@@ -34,7 +34,7 @@ __tf_collect_if_needed() {
     esac
     $will || return 0
     mkdir -p "$artifact_dir/$stamp"
-    cp -r $tmpdir/* "$artifact_dir/$stamp"
+    cp -r "$tmpdir"/* "$artifact_dir/$stamp"
 }
 
 __tf_default_header() {
@@ -65,9 +65,9 @@ tf_enum_tests() {
     # List what looks like test; relative to $TF_SUITE
     #
     tf_debug "TF_SUITE='$TF_SUITE'"
-    test -d $TF_SUITE || return 0
+    test -d "$TF_SUITE" || return 0
     find -L \
-        $TF_SUITE \
+        "$TF_SUITE" \
         -mindepth 2 \
         -maxdepth 2 \
         -type f \
@@ -95,16 +95,16 @@ tf_run_tests() {
     do
         tf_think "... $tname"
         tmpdir=$(mktemp -d)
-        stamp=$(date +artifacts-$tname-%Y%m%d-%H%M%S)
+        stamp=$(date "+artifacts-$tname-%Y%m%d-%H%M%S")
         cp -r "$TF_SUITE/$tname/"* "$tmpdir"
-        pushd $tmpdir >/dev/null
+        pushd "$tmpdir" >/dev/null
             TF_DIR="$tf_dir" TF_SUITE=$tf_suite TF_TNAME="$tname" \
                 ./TF_RUN
             tes=$?
             __tf_collect_if_needed $tes
             test $tes -gt $es && es=$tes
         popd >/dev/null
-        rm -rf $tmpdir
+        rm -rf "$tmpdir"
         if test $tes -eq 0;
         then
             tf_think "''' $tname ($tes)"

utils/tfkit/include/subtest.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-. $TF_DIR/include/common.sh
+. "$TF_DIR/include/common.sh"
 
 tf_enum_subtests() {
     #
@@ -22,7 +22,7 @@ tf_do_subtest() {
     #
     # Run single subtest inc. setup/cleanup if present
     #
-    local subtname=$1       # this subtest name
+    local subtname="$1"     # this subtest name
     local ses=0             # subtest exit status
     local tcmd=""           # test command
     local setup=true        # setup command
@@ -31,7 +31,7 @@ tf_do_subtest() {
     test -f TF_CLEANUP && cleanup=". TF_CLEANUP"
     if $setup;
     then
-        tcmd="$(tf_name2cmd $subtname)"
+        tcmd="$(tf_name2cmd "$subtname")"
         tf_debug "tcmd='$tcmd'"
         $tcmd; ses=$?
     else

utils/tfkit/include/tools.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-. $TF_DIR/include/common.sh
+. "$TF_DIR/include/common.sh"
 
 # 1. exec: [test] -> [result]
 # 2. eval:           [result] == [oracle]
@@ -30,17 +30,21 @@ tf_testflt() {
 
     # get args
     #
+    local orig_args="$0 $*"
+    tf_debug "orig_args=$orig_args"
+    local arg_err=false
     while true; do case "$1" in
-        -i) t_in="$2";          shift 2 ;;
-        -n) t_name="$2";        shift 2 ;;
-        -O) o_out="$2";         shift 2 ;;
-        -E) o_err="$2";         shift 2 ;;
-        -S) o_es="$2";          shift 2 ;;
+        -i) t_in="$2";          shift 2 || { arg_err=true; break; } ;;
+        -n) t_name="$2";        shift 2 || { arg_err=true; break; } ;;
+        -O) o_out="$2";         shift 2 || { arg_err=true; break; } ;;
+        -E) o_err="$2";         shift 2 || { arg_err=true; break; } ;;
+        -S) o_es="$2";          shift 2 || { arg_err=true; break; } ;;
         --)                     shift; break ;;
         "")                            break ;;
         -*) tf_warn "wrong testcli arg: $1"; return $TF_ES_BAILOUT ;;
         *)                             break ;;
     esac done
+    $arg_err && { tf_warn "error parsing arguments: $orig_args"; return $TF_ES_BAILOUT; }
     tf_debug "t_in='$t_in'"
     tf_debug "t_name='$t_name'"
     tf_debug "o_out='$o_out'"
@@ -53,6 +57,7 @@ tf_testflt() {
     test -r "$t_in"    || { tf_warn "missing input file: $t_in"     ; return $TF_ES_BAILOUT; }
     test -e "$o_out"   || { tf_warn "missing oracle stdout: $o_out" ; return $TF_ES_BAILOUT; }
     test -e "$o_err"   || { tf_warn "missing oracle stderr: $o_err" ; return $TF_ES_BAILOUT; }
+    test "$o_es" -ge 0 || { tf_warn "invalid oracle status: $o_es"  ; return $TF_ES_BAILOUT; }
 
     # prepare
     #
@@ -61,18 +66,18 @@ tf_testflt() {
     r_err="result/$t_name.stderr"
     tf_debug "r_out='$r_out'"
     tf_debug "r_err='$r_err'"
-    touch $r_out || { tf_warn "cannot create tmp file: $r_out" ; return $TF_ES_BAILOUT; }
-    touch $r_err || { tf_warn "cannot create tmp file: $r_err" ; return $TF_ES_PANIC; }
+    touch "$r_out" || { tf_warn "cannot create tmp file: $r_out" ; return $TF_ES_BAILOUT; }
+    touch "$r_err" || { tf_warn "cannot create tmp file: $r_err" ; return $TF_ES_PANIC; }
 
     # run
     #
-    ( <$t_in eval "$@" >$r_out 2>$r_err ); r_es=$?
+    ( <"$t_in" eval "$@" >"$r_out" 2>"$r_err" ); r_es=$?
     tf_debug "r_es='$r_es'"
 
     # eval/report/exit
     #
     test $r_es = $o_es || { tf_warn "bad exit status: $r_es (need $o_es)" ; t_es=$TF_ES_FAIL; }
-    diff -u $o_err $r_err || t_es=$TF_ES_FAIL
-    diff -u $o_out $r_out || t_es=$TF_ES_FAIL
+    diff -u "$o_err" "$r_err" || t_es=$TF_ES_FAIL
+    diff -u "$o_out" "$r_out" || t_es=$TF_ES_FAIL
     return $t_es
 }

utils/tfkit/runtests

@@ -2,6 +2,7 @@
 # ffoo test framework
 # See LICENSE file for copyright and license details.
 
+export TF_VERSION="0.0.2"
 
 die() {
     echo "$@" && exit 9
@@ -13,15 +14,17 @@ export TF_SUITE="${TF_SUITE:-tests}"
 export TF_ARTIFACTS="${TF_ARTIFACTS:-tfkit-artifacts}"
 export TF_COLLECT="${TF_COLLECT:-auto}"
 
-. $TF_DIR/include/harness.sh \
- || die "cannot import harness; is TF_DIR set properly?: $TF_DIR"
-
 
 usage() {
-    echo "usage: $(basename $0) [-c|-C] [-t tests_re] [-s subtest_re] [-p binpath] [-v] [-d]" >&2
+    echo "usage: $(basename "$0") [-c|-C] [-t tests_re] [-s subtest_re] [-p binpath] [-v] [-d]" >&2
     exit 2
 }
 
+version() {
+    echo "TFKit (A trivial test kit) $TF_VERSION"
+    exit 0
+}
+
 while true; do case "$1" in
     -c|--collect)           TF_COLLECT=always; shift ;;
     -C|--no-collect)        TF_COLLECT=never; shift ;;
@@ -30,8 +33,13 @@ while true; do case "$1" in
     -p|--prefix)            export PATH="$(readlink -f "$2")/bin:$PATH"; shift 2 ;;
     -d|--debug)             export TF_DEBUG=true; shift ;;
     -v|--verbose)           export TF_VERBOSE=true; shift ;;
+    --version)              version ;;
+    --version-semver)       echo "$TF_VERSION"; exit 0 ;;
     "") break ;;
     *)  usage ;;
 esac done
 
+. "$TF_DIR/include/harness.sh" \
+ || die "cannot import harness; is TF_DIR set properly?: $TF_DIR"
+
 time tf_run_tests