Initial commit

Makefile

@@ -0,0 +1,4 @@
+# See LICENSE.md file for copyright and license details.
+
+MKIT_DIR=/usr/share/jattool/tdk/utils/mkit
+include $(MKIT_DIR)/mkit.mk

README.md

@@ -0,0 +1,152 @@
+preupgrade-assistant JAT test suite
+===================================
+
+This is an independent test suite for preupgrade-assistant that makes use
+of JATS - Just A Test System.
+
+JATS defines test suite format, which is a directory tree consisting of
+individual tests.  These individual tests can in principle be implemented
+in any language, but for now *shellfu-bash-jat* is the only available
+library.
+
+
+Dependencies - test machine
+---------------------------
+
+All specific dependencies are available in [amahdal/jats][aj] COPR project.
+However, since preupgrade-assistant itself is only available for RHEL6,
+which does not have native COPR support, you will need to add repositories
+manually.
+
+  [aj]: https://copr.fedorainfracloud.org/coprs/amahdal/jats/
+
+On the testing machine, you will need:
+
+ *  shellfu-bash-jat - the main JAT library,
+ *  shellfu-bash-preupg - preupgrade-assistant test library,
+ *  jattool - harness, exploration and reporting toolkit
+ *  jats-preupgrade-assistant - this test suite
+
+Note that tests don't formally depend on preupgrade-assistant itself, so you will
+need to install it manually. (This is to help you retain full control of the
+preupgrade-assistant version; in future there will be mechanisms to help with that
+but for now manual is the safest.).
+
+Also note that you don't need to install all sub-components of jattool: jattool-tdk
+and jattool-report require things not available easily on all RHEL versions and are
+normally only useful on workstations anyway.
+
+Last but not least: you don't really need to install the jats-preupgrade-assistant;
+you could do away with just cloning this repo.  However, RPM will help you with
+the dependencies, so it's advised to install it even if you want to run from the
+repo.
+
+
+Running tests
+-------------
+
+At this point, most tests should work without installation.   That is, just
+clone this repo and run
+
+    jattool tfind
+
+to see lists of tests.   Each JAT test is a directory, so above should give
+you list of directories: you can then run any of them using `jattool runtest`,
+eg.:
+
+    jattool runtest ./src/api/binary_req
+
+
+Dependencies - worksation
+-------------------------
+
+Since your workstation is probably Fedora, you can use the modern ways:
+
+    dnf copr enable amahdal/jats
+    dnf install shellfu-bash-jat jattool jattool-report jattool-tdk
+                #\
+        # don't be put off of the number of dependencies; let's just say
+        # that author of Shellfu loves granularity :-)
+
+This will give you basic "test development kit".  More on that later.
+
+
+Exploring (and hacking on) tests
+--------------------------------
+
+I won't go into details; I'll tell you just enough to get you started,
+but be aware that **(mostly) everything is (mostly) documented** and
+primary ways to get help (actually more like reference, but helps)
+using *sfdoc* tool, which should be on your worksation by now.
+
+Tests normally only consist of one important file, named ... you
+guessed it...  *test*.  (There's also *main.fmf*, but that only contains
+meta-data, which is not even really used these days.)
+
+    $ tree src/api/binary_req
+    src/api/binary_req
+    ├── main.fmf
+    └── test
+
+So open that file:  it's a Bash script with extra "enhancements":  you'll see things
+like:
+
+    shellfu import jat
+    shellfu import preupg
+    shellfu import ...
+    shellfu import xcase
+
+Shellfu platform brings modularity to Bash, so the names above are names
+of modules.  `jat` is a module name.  And you can get reference on the module
+
+    sfdoc jat
+
+Since this is Bash, modules contain just functions and variables.  And since
+Bash has no namespaces, an important convention for (almost) all modules
+is in place:
+
+ *  any functions defined by module `foo` must be named by `foo__` prefix,
+ *  any variables defined by module `foo` must be named by `FOO__` prefix.
+
+Eg. an initializing function from module *bar* would have to be called
+`bar__init()`.
+
+Yep, this is a bit annoying, but:
+
+ *  you'll get used,
+
+ *  since Bash has no namespaces, this is the only way to avoid conflicts
+    without having to read every imported module's source code (recursively,
+    and after every update).
+
+ *  it can actually be helpful: seeing function like `jat__eval`, you
+    *immediately know* it's from *jat* module (heck, you can call `sfdoc
+    -O jat__eval`),
+
+ *  the above also applies to possible tools you might want to develop (it's
+    possible to *parse* module name from object name),
+
+ *  it provides nice excuse to keep module and function names short!
+
+
+Creating new tests
+------------------
+
+**NOTE:** At the time of this writing, the test creation kit is not finished,
+so if you really need new test, you can just copy another one.
+
+In order to avoid duplication of work and unnecessary frustration,
+take following steps:
+
+ 1. Before you start, **contact maintainer** and inform them that you
+    are planning to write a test.  You can do this by filing an issue
+    in Github repo or by contacting the maintainer directly.
+
+ 2. Read [Shellfu coding style][sfcs]; tests need to follow this.
+
+ 3. Be aware that the JATS project is in early stage (`v0.0.*`), and
+    some (most?) pieces of coding style and conventions are not yet
+    written down.  A more consistent documentation is expected to arrive
+    before `v0.1.0`.
+
+  [sfcs]: https://github.com/AloisMahdal/shellfu/blob/last/notes/style.md

mkit.ini

@@ -0,0 +1,17 @@
+[INCLUDE:/usr/share/jattool/tdk/skel/mkit.ini]
+
+[project]
+    pkgname    = jats-preupgrade-assistant
+    version    = 0.0.0
+    maintainer = Alois Mahdal <amahdal@redhat.com>
+
+[macros]
+    __JATS_SUITENAME__ = preupgrade-assistant
+    __JATS_REQUIRES__ = shellfu-bash-preupg
+    __JATS_REQUIRES__ = shellfu-bash-xcase
+    __JATS_REQUIRES__ = libxml2
+
+[files]
+    jats = src preupgrade-assistant
+
+#mkit version=0.0.32

src/.jats/domain

@@ -0,0 +1 @@
+redhat.com

src/.jats/format

@@ -0,0 +1 @@
+0.0

src/.jats/ns

@@ -0,0 +1 @@
+oamg

src/.jats/sut_pname

@@ -0,0 +1 @@
+preupgrade-assistant

src/.jats/version.skel

@@ -0,0 +1 @@
+__MKIT_PROJ_VERSION__

src/api/binary_req/main.fmf

@@ -0,0 +1,2 @@
+description: binary_req auto-check
+timeout: 90m

src/api/binary_req/test

@@ -0,0 +1,163 @@
+#!/bin/bash
+
+shellfu import jat
+shellfu import jat_dump
+shellfu import preupg
+shellfu import preupg_fupath
+shellfu import xcase
+
+T_MESSAGE=
+T_RISKTXT='Please, install all required packages (and binaries) and run preupg again to process check properly.'
+#shellcheck disable=SC2034
+PREUPG__RULE=xccdf_preupg_rule_api_binary_req_check
+
+assert_irisknum() {
+    jat__cmp -h "number of per-item risks is $1" \
+        "$(wc -l <irisks)" eq "$1"
+}
+
+mkbrq() {
+    #
+    # Produce the binary_req INI key/value pair
+    #
+    local head=true
+    test -n "$bins" || return 0
+    echo -n "binary_req = "
+    for bin in "${bins[@]}";
+    do
+        $head || echo -n ', '
+        case $bin in
+            mandatory)  echo -n "rpm" ;;
+            impossible) echo -n "nonexistentbin$(rtoken)" ;;
+        esac
+        head=false
+    done
+    echo
+}
+
+rtoken() {
+    head -c 1000 /dev/urandom \
+      | md5sum \
+      | head -c 7
+}
+
+XCASE__ARRAYS=bins
+
+xcase__enum() {
+    #
+    # Enumerate subtest ids
+    #
+    # Variables:
+    #
+    #  *  lang - language used: `py` for Python, `sh` for Bash
+    #
+    #  *  bins - binary "availability classes" used for testing.  This defines
+    #            what kind of binary name will be added to binary_req.  Multiple
+    #            classes mean there will be multiple binary names.
+    #
+    #            `mandatory` means a binary such as "rpm" that is impossible to
+    #            *not* be on the system.  `impossible` means a randomly generated
+    #            binary name.
+    #
+    #
+
+    for lang in py sh;
+    do
+        echo lang=$lang,bins=mandatory
+        echo lang=$lang,bins=impossible
+        echo lang=$lang,bins=
+        echo lang=$lang,bins=mandatory+mandatory
+        echo lang=$lang,bins=mandatory+impossible
+        echo lang=$lang,bins=impossible+mandatory
+        echo lang=$lang,bins=impossible+impossible
+    done
+
+}
+
+#shellcheck disable=SC2154
+xcase__setup() {
+    #
+    # Prepare for subtest
+    #
+    T_MESSAGE=message$(rtoken)
+    {
+        echo '[preupgrade]'
+        mkbrq
+        echo "[MODULE]"
+        echo "GROUP = api"
+        echo "NAME = binary_req"
+        echo "LANG = $lang"
+        case $lang in
+            py)
+                echo "CODE = log_info('$T_MESSAGE')"
+                echo "CODE = exit_informational()"
+                ;;
+            sh)
+                echo "CODE = log_info $T_MESSAGE"
+                echo "CODE = exit_informational"
+                ;;
+        esac
+    } > module.ini
+    preupg_fupath RHEL6_7 module.ini
+    jat__cmd -h "work around BZ#1368823" \
+        cp -ar RHEL6_7 RHEL6_7-backup
+    PREUPG__UPATH="RHEL6_7/all-xccdf.xml" \
+        preupg__run1
+    preupg__get_messages > messages
+    preupg__get_risks > risks
+    grep 'HIGH: Binary .* is not installed' risks > irisks
+}
+
+xcase__test() {
+    #
+    # Do the work
+    #
+    local o_behavior=ok     # behavior class
+    local o_irisknum=0      # per-item risks
+
+    for bin in "${bins[@]}"
+    do
+        case $bin in
+            impossible)
+                o_behavior=fail
+                ((o_irisknum++))
+                ;;
+        esac
+    done
+
+    case $o_behavior in
+        ok)
+            preupg__assert    \
+                   result "informational"   \
+                -N risk.high "$T_RISKTXT"   \
+                   msg.info "$T_MESSAGE"
+            assert_irisknum 0
+            ;;
+        fail)
+            preupg__assert    \
+                   result "needs_action"   \
+                   risk.high "$T_RISKTXT"   \
+                -N msg.info "$T_MESSAGE"
+            assert_irisknum $o_irisknum
+            ;;
+    esac
+
+}
+
+xcase__diag() {
+    #
+    # Burp up some diag
+    #
+    jat_dump__file module.ini risks messages irisks
+}
+
+xcase__cleanup() {
+    #
+    # Clean up after subtest
+    #
+    preupg__rmresult
+}
+
+
+xcase__run -v
+preupg__Cleanup

src/api/deploy_hook/main.fmf

@@ -0,0 +1,2 @@
+description: hook deployment
+timeout: 90m

src/api/deploy_hook/test

@@ -0,0 +1,272 @@
+#!/bin/bash
+
+shellfu import jat_dump
+shellfu import preupg
+shellfu import preupg_fupath
+shellfu import xcase
+
+PACKAGE="preupgrade-assistant"
+#shellcheck disable=SC2034
+PREUPG__RULE=xccdf_preupg_rule_api_deploy_hook_check
+HOOKS_ROOT="/root/preupgrade/hooks"
+HOOK_DIRNAME="api_deploy_hook"
+
+# mute ShellCheck
+Beh=
+CallSeq=
+
+assert_deployed() {
+    #
+    # Assert our hooks were deployed
+    #
+    local call
+    local arg
+    local hookdir
+    for call in "${CallSeq[@]}";
+    do
+        debug -v call
+        case $call in
+            pre.*)  hookdir="$HOOKS_ROOT/$HOOK_DIRNAME/preupgrade" ;;
+            post.*) hookdir="$HOOKS_ROOT/$HOOK_DIRNAME/postupgrade" ;;
+            #FIXME: structure unclear at the time of this writing
+            *)  xcase__id_error CallSeq
+        esac
+        jat__cmd -h "hook dir exists" \
+            test -d "$hookdir"
+        for arg in $(tr . '\n' <<<"${call#*.}");
+        do
+            debug -v arg
+            case $arg in
+                s*|as*)
+                    jat__cmd -h "hook script content is correct" \
+                        grep "echo hi I am script_$arg.sh" "$hookdir/run_hook"
+                    ;;
+                f*|af*)
+                    jat__cmd -h "auxiliary file content is correct" \
+                        grep "text of file_$arg" "$hookdir/file_$arg.conf"
+                    ;;
+                d*)
+                    jat__cmd -h "file_a from auxiliary dir is present" \
+                        test -s "$hookdir/dir_$arg/file_a"
+                    jat__cmd -h "file_b from auxiliary dir is present" \
+                        test -s "$hookdir/dir_$arg/file_b"
+                    ;;
+                *)
+                    warn "assert_deployed() arg unhandled by assert: $arg"
+                    ;;
+            esac
+        done
+    done
+}
+
+rtoken() {
+    head -c 1000 /dev/urandom \
+      | md5sum \
+      | head -c 7
+}
+
+mktree() {
+    #
+    # Create tree (ini for fupath) for deployment
+    #
+    local call
+    local arg
+    for call in "${CallSeq[@]}";
+    do
+        for arg in ${call//./ };
+        do
+            debug -v arg
+            case $arg in
+                post|pre|badtype|X) true                                            ;;
+                as*)    echo "echo hi I am script_$arg.sh; find" >"script_$arg.sh"  ;;
+                af*)    echo "text of file_$arg" >"file_$arg.conf"                  ;;
+                s*)     echo "script_$arg.sh = echo hi I am script_$arg.sh; find"   ;;
+                f*)     echo "file_$arg.conf = text of file_$arg"                   ;;
+                d*)     echo "dir_$arg/file_a = text of dir_$arg/file_a"
+                        echo "dir_$arg/file_b = text of dir_$arg/file_b"            ;;
+                *)      xcase__id_error CallSeq ;;
+            esac
+        done
+    done
+}
+
+mkcalls() {
+    #
+    # Create deploy_hook call (ini for fupath)
+    #
+    local call
+    local arg
+    for call in "${CallSeq[@]}";
+    do
+        debug -v call
+        echo -n "CODE = deploy_hook"
+        case $lang:$call in
+            py:pre.*)      echo -n "('preupgrade'" ;;
+            py:post.*)     echo -n "('postupgrade'" ;;
+            py:badtype.*)  echo -n "('badtype'" ;;
+            sh:pre.*)      echo -n " preupgrade" ;;
+            sh:post.*)     echo -n " postupgrade" ;;
+            sh:badtype.*)  echo -n " badtype" ;;
+            *) xcase__id_error lang CallSeq ;;
+        esac
+        for arg in $(tr . '\n' <<<"${call#*.}");
+        do
+            debug -v arg
+            case $lang:$arg in
+                py:as*)    echo -n ",'$PWD/script_$arg.sh'" ;;
+                py:af*)    echo -n ",'$PWD/file_$arg.conf'" ;;
+                py:s*)     echo -n ",'script_$arg.sh'" ;;
+                py:f*)     echo -n ",'file_$arg.conf'" ;;
+                py:d*)     echo -n ",'dir_$arg'" ;;
+                py:X)      echo -n ",'nonexistent_file_$(rtoken)'" ;;
+                sh:as*)    echo -n " $PWD/script_$arg.sh" ;;
+                sh:af*)    echo -n " $PWD/file_$arg.conf" ;;
+                sh:s*)     echo -n " script_$arg.sh" ;;
+                sh:f*)     echo -n " file_$arg.conf" ;;
+                sh:d*)     echo -n " dir_$arg" ;;
+                sh:X)      echo -n " nonexistent_file_$(rtoken)" ;;
+                *) xcase__id_error lang CallSeq ;;
+            esac
+        done
+        case $lang in
+            py) echo ')';;
+            sh) echo    ;;
+        esac
+    done
+    echo "CODE = exit_fixed$(test "$lang" = py && echo '()')"
+}
+
+
+XCASE__ARRAYS=CallSeq
+
+xcase__enum() {
+    #
+    # Enumerate subtest ids
+    #
+
+    # Parameters are 'Beh', behavior hint (to help simplify test phase),
+    # and 'CallSeq' - plus-separated list of deploy_hook() calls in this
+    # form:
+    #
+    #     TYPE[.ARG][.ARG]..
+    #
+    # where TYPE is either 'post' or 'pre', and ARG are symbols for the
+    # arguments deploy_hook() is called with:
+    #
+    #  *   sN means script N (1, 2, ...),
+    #  *   fN means file N (1, 2, ...),
+    #  *   asN and afN mean the same as sN and fN, but will be called by
+    #      absolute path
+    #  *   dN means directory N (1, 2, ...),
+    #  *   X means non-existent file
+    #
+    # Actual module code as well as all necessary files are created during
+    # setup phase.
+    #
+
+    {
+
+        # normal cases
+        #
+
+        # simplest
+        echo Beh=OK,CallSeq=post.s1
+        echo Beh=OK,CallSeq=post.s1.f1
+        echo Beh=OK,CallSeq=post.s1.f1.d1
+        echo Beh=OK,CallSeq=pre.s1
+        echo Beh=OK,CallSeq=pre.s1.f1
+        echo Beh=OK,CallSeq=pre.s1.f1.d1
+        echo Beh=OK,CallSeq=pre.as1
+        echo Beh=OK,CallSeq=pre.s1.af1
+
+        # pre and post
+        echo Beh=OK,CallSeq=pre.s1+post.s2
+        echo Beh=OK,CallSeq=pre.s1.f1.d1+post.s2.f2.d2
+        echo Beh=OK,CallSeq=post.s1.f1.d1+pre.s2.f2.d2
+
+        # bad cases
+        #
+
+        # deploy twice: same, smaller, bigger
+        echo Beh=BAD,CallSeq=post.s1+post.s1
+        echo Beh=BAD,CallSeq=post.s1.f1.d1+post.s1
+        echo Beh=BAD,CallSeq=post.s1+post.s1.f1.d1
+
+        # bad args: no script, script is dir, nonexistent, double arg, bad type
+        echo Beh=BAD,CallSeq=post.
+        echo Beh=BAD,CallSeq=post.d1
+        echo Beh=BAD,CallSeq=post.X
+        echo Beh=BAD,CallSeq=post.X.X
+        echo Beh=BAD,CallSeq=post.s1.X.X
+#       echo Beh=BAD,CallSeq=post.s1.f1.f1      # not sure if we can/need to catch this
+        echo Beh=BAD,CallSeq=post.s1.d1.X
+        echo Beh=BAD,CallSeq=badtype.s1
+
+    } | xcase__per lang sh py
+}
+
+#shellcheck disable=SC2154
+xcase__setup() {
+    #
+    # Prepare for subtest
+    #
+    {
+        echo "[MODULE]"
+        echo "GROUP = api"
+        echo "NAME = deploy_hook"
+        echo "LANG = $lang"
+        mkcalls
+        echo "[FILES]"
+        mktree
+    } > module.ini
+    preupg_fupath RHEL6_7 module.ini
+    PREUPG__UPATH="RHEL6_7/all-xccdf.xml" \
+        preupg__run1 --skip-common
+}
+
+xcase__test() {
+    #
+    # Do the work
+    #
+    jat__eval -S 0,1 -h "store hook file list" \
+        "find '$HOOKS_ROOT' -type f | grep deploy_hook > hooks"
+    case $Beh in
+
+        OK)
+            assert_deployed
+            jat__eval -S 1 -h "no errors were logged" \
+                "preupg__get_messages ERROR | grep ."
+            ;;
+
+        BAD)
+            jat__eval -S 0 -h "error was logged" \
+                "preupg__get_messages ERROR | grep ."
+            preupg__assert result 'error'
+            ;;
+
+        *)
+            xcase__id_error Beh
+            ;;
+    esac
+}
+
+xcase__diag() {
+    #
+    # Burp up some diag
+    #
+    jat_dump__file module.ini hooks
+    preupg__get_node | jat_dump__pipe NODE
+    preupg__get_messages | jat_dump__pipe LOG_MESSAGES
+}
+
+xcase__cleanup() {
+    #
+    # Clean up after subtest
+    #
+    preupg__rmresult
+}
+
+PREUPG__UPATH=@pass \
+    preupg__Run1
+xcase__run -v
+preupg__Cleanup

src/api/logging/main.fmf

@@ -0,0 +1,2 @@
+description: logging facilities
+timeout: 1h

src/api/logging/test

@@ -0,0 +1,106 @@
+#!/bin/bash
+
+shellfu import preupg
+shellfu import preupg_fupath
+shellfu import xcase
+shellfu import jat_dump
+
+#shellcheck disable=SC2034
+PREUPG__RULE=xccdf_preupg_rule_api_logging_check
+
+
+assert_msg_like() {
+    #
+    # Assert log message matching $1 exists
+    #
+    preupg__get_messages > msgs
+    jat__cmd grep "$1" msgs
+}
+
+xcase__enum() {
+    #
+    # Enumerate subtest ids
+    #
+    xcase__permute lang    py sh \
+      | xcase__per fn      log_debug log_info log_warning log_error \
+      | xcase__per debug   true false
+}
+
+#shellcheck disable=SC2154
+xcase__setup() {
+    #
+    # Prepare for subtest
+    #
+    local maybe_debug=""
+    $debug && maybe_debug="--debug"
+    echo "message$(head -c 1000 /dev/urandom | md5sum | head -c 7)" >t_message
+    {
+        echo "[MODULE]"
+        echo "GROUP = api"
+        echo "NAME = logging"
+        echo "LANG = $lang"
+        case $lang in
+            sh)
+                echo "CODE = $fn $(<t_message)"
+                echo "CODE = exit_informational"
+                ;;
+            py)
+                echo "CODE = $fn('$(<t_message)')"
+                echo "CODE = exit_informational()"
+                ;;
+        esac
+    } > module.mdef
+    preupg_fupath RHEL6_7 module.mdef
+    PREUPG__UPATH="RHEL6_7/all-xccdf.xml" \
+        preupg__run1 $maybe_debug --skip-common
+}
+
+xcase__test() {
+    #
+    # Do the work
+    #
+
+    # set oracles
+    #
+    local o_severity=""
+    local o_msgnum=0
+    local o_result=informational
+    local o_not=""
+
+    case $fn:$debug in
+        log_debug:true)   o_msgnum=1; o_severity=DEBUG    ;;
+        log_debug:false)  o_msgnum=0; o_severity=DEBUG    ;;
+        log_info:*)       o_msgnum=1; o_severity=INFO     ;;
+        log_warning:*)    o_msgnum=1; o_severity=WARNING  ;;
+        log_error:*)      o_msgnum=1; o_severity=ERROR    ;;
+    esac
+    test "$o_msgnum" -eq 0 && o_not=-N
+
+    # asserts
+    #
+    preupg__assert \
+        result "$o_result"  \
+         msgn "$o_msgnum"   \
+         $o_not msg.$o_severity "$(<t_message)"
+}
+
+xcase__diag() {
+    #
+    # Burp up some diag
+    #
+    jat_dump__file module.mdef
+    preupg__get_messages | jat_dump__pipe MESSAGES
+}
+
+xcase__cleanup() {
+    #
+    # Clean up after subtest
+    #
+    preupg__rmresult
+}
+
+
+PREUPG__UPATH=@pass \
+    preupg__Run1
+xcase__run -v
+preupg__Cleanup

src/api/mseverity/main.fmf

@@ -0,0 +1,2 @@
+description: module result severity
+timeout: 90m

src/api/mseverity/test

@@ -0,0 +1,310 @@
+#!/bin/bash
+
+shellfu import jat_dump
+shellfu import preupg
+shellfu import preupg_fupath
+shellfu import xcase
+
+#shellcheck disable=SC2034
+{
+    PREUPG__RULE="xccdf_preupg_rule_api_severity_check"
+    PREUPG__UPATH="RHEL6_7/all-xccdf.xml"
+    PREUPG__VALID_ES='0-15'
+    T_MSCRIPT="RHEL6_7/api/severity/check"
+}
+
+
+# ...................... #                                                     #
+# CASE GENERATOR HELPERS # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #
+# '''''''''''''''''''''' #                                                     #
+
+dec_event() {
+    #
+    # Decode numerical event id
+    #
+    case $1 in
+
+        # the prefix is collation class decorator; gen_eseq will
+        # sort|uniq the result, so here you can define forced
+        # order.  (Due to uniq, this also means that events in same
+        # class will be "squashed" into one if found.)
+
+        # log messages
+        #
+        0) echo ;;
+        1) echo A.log_debug;;
+        2) echo A.log_info ;;
+        3) echo A.log_error ;;
+        4) echo A.log_warning ;;
+
+        # risks
+        #
+        10) echo ;;
+        11) echo A.log_slight_risk ;;
+        12) echo A.log_medium_risk ;;
+        13) echo A.log_high_risk ;;
+        14) echo A.log_extreme_risk ;;
+
+        # exits
+        #
+        20) echo ;;
+        21) echo Z.exit_pass ;;
+        22) echo Z.exit_fail ;;
+        23) echo Z.exit_error ;;
+        24) echo Z.exit_not_applicable ;;
+        25) echo Z.exit_informational ;;
+        26) echo Z.exit_fixed ;;
+    esac
+}
+
+echoln() {
+    #
+    # Echo each argument separately
+    #
+    for ln in "$@";
+    do
+        echo "$ln"
+    done
+}
+
+mkesg() {
+    #
+    # Make event sequence generator
+    #
+    local patt=$1
+    local eclass    # event class
+    echo -n 'echoln '
+    grep -o . <<<"$patt" \
+      | while read -r eclass;
+        do
+            case $eclass in
+                m)  echo -n 'm{0..4}' ;;
+                r)  echo -n 'm{10..14}' ;;
+                x)  echo -n 'x{20..28}' ;;
+            esac
+        done
+}
+
+gen_eseq() {
+    #
+    # Get all possible programs in pattern $1
+    #
+    # Usage: gen_eseq PATT
+    #
+    # The PATT has format C[C..], where C is a "event class" and can
+    # be one of following:
+    #
+    #     'm' for log message,
+    #     'r' for risk log,
+    #     'x' for exit_* call,
+    #
+    # Result is all possible unique event sequences (one per line)
+    # conforming to PATT, where:
+    #
+    #  *  order between 'm' and 'r' is not preserved,
+    #  *  'x' is always last,
+    #  *  each event can happen at least zero times and at most that
+    #     many times how many times the class was seen.
+    #
+    # Few examples to shed some light.  Programs generated by following
+    # PATTs may have certain properties:
+    #
+    #  *  'mrx' can have 0-1 log messages, 0-1 risks and 0-1 exits,
+    #  *  'mmrx' can have 0-2 log messages, 0-1 risks and 0-1 exits
+    #  *  'mrmx' is the same because log/risk order is not significant,
+    #  *  'mrxm' is still the same because 'x' is forced to be last,
+    #  *  'mrxx' would have 0-2 exit calls,
+    #
+    # Note that these are not real programs but just lists of events
+    # (named as API functions just for convenience); you will have to
+    # parse the list (comma-separated) and append arguments where needed.
+    #
+    local patt=$1   # event pattern
+    local eseq      # event id sequence
+    local eid       # event id
+    local esg       # sequence generator (makes use of '{..}' Bash syntax)
+    esg=$(mkesg "$patt")
+    eval "$esg" \
+      | while read -r eseq;
+        do
+            grep -oE '[[:alpha:]][[:digit:]]+' <<<"$eseq" \
+              | while read -0r eid;
+                do
+                    dec_event "${eid:1}"
+                done \
+              | grep . \
+              | sort \
+              | uniq \
+              | colrm 1 2 \
+              | paste -sd,
+        done \
+          | grep . \
+          | sort \
+          | uniq
+}
+
+
+# ............. #                                                              #
+# SETUP HELPERS # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #
+# ''''''''''''' #                                                              #
+
+mkcode() {
+    #
+    # Make body of module
+    #
+    xcase__id \
+      | tr , '\n' \
+      | while read -r event;
+        do
+            case $event in
+                log_*_risk) echo "$event risk.by.$event" ;;
+                log_*)      echo "$event message.from.$event" ;;
+                exit_*)     echo "$event" ;;
+            esac
+        done
+}
+
+
+# ............. #                                                              #
+# INTROSPECTION # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #
+# ''''''''''''' #                                                              #
+
+risksort() {
+    #
+    # Sort risk log calls based on severity (decorated sort)
+    #
+    local call      # the risk call
+    local args      # ... args (finally ignored)
+    while read -r call args;
+    do
+        case $call in
+            log_slight_risk)    echo -n '1 ' ;;
+            log_medium_risk)    echo -n '2 ' ;;
+            log_high_risk)      echo -n '3 ' ;;
+            log_extreme_risk)   echo -n '4 ' ;;
+        esac
+        echo "$call $args"
+    done \
+      | sort -n \
+      | cut -d' ' -f2
+}
+
+logmsort() {
+    #
+    # Sort log message calls based on severity (decorated sort)
+    #
+    local call      # the log call
+    local args      # ... args (finally ignored)
+    while read -r call args;
+    do
+        case $call in
+            log_debug)      echo -n '1 ' ;;
+            log_info)       echo -n '2 ' ;;
+            log_warning)    echo -n '3 ' ;;
+            log_error)      echo -n '4 ' ;;
+        esac
+        echo "$call $args"
+    done \
+      | sort -n \
+      | cut -d' ' -f2
+}
+
+get_worst_risk() {
+    grep '^log_[a-z]*_risk' "$T_MSCRIPT" \
+      | risksort \
+      | tail -1
+}
+
+get_first_exit() {
+    grep ^exit_ "$T_MSCRIPT" \
+      | head -1
+}
+
+get_risknum() {
+    grep -c '^log_[a-z]*_risk' "$T_MSCRIPT"
+}
+
+get_worst_logm() {
+    grep '^log_' "$T_MSCRIPT" \
+      | grep -v '_risk\b' \
+      | logmsort \
+      | tail -1
+}
+
+
+# .............. #                                                             #
+# XCASE HANDLERS # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #
+# '''''''''''''' #                                                             #
+
+xcase__enum() {
+    gen_eseq rx
+    #FIXME: add larger modes such as 'rrx', 'mrx' or 'rxx'
+}
+
+
+xcase__setup() {
+    {
+        echo '[MODULE]'
+        echo 'GROUP = api'
+        echo 'NAME = severity'
+        mkcode | sed 's/^/CODE = /'
+    } > module.mdef
+    preupg_fupath RHEL6_7 "module.mdef"
+}
+
+xcase__test() {
+    local o_es          # oracle: exit status
+    local o_result      # oracle: assessment result
+
+    # set up oracles
+    #
+    case $(get_first_exit):$(get_worst_risk) in
+        # obviously error
+        #
+        :*)                         o_es=2; o_result=error ;;
+        exit_error:*)               o_es=2; o_result=error ;;
+
+        # valid calls to exit_fail
+        #
+        exit_fail:log_slight_risk)  o_es=0; o_result=needs_inspection ;;
+        exit_fail:log_medium_risk)  o_es=0; o_result=needs_inspection ;;
+        exit_fail:log_high_risk)    o_es=1; o_result=needs_action ;;
+        exit_fail:log_extreme_risk) o_es=2; o_result=fail ;;
+
+        # (mostly in)valid calls to other exit_*
+        #
+        exit_pass:*)                o_es=0; o_result=pass ;;
+        exit_not_applicable:*)      o_es=0; o_result=notapplicable ;;
+        exit_informational:*)       o_es=0; o_result=informational ;;
+        exit_fixed:*)               o_es=0; o_result=fixed ;;
+        exit_fail:)                 o_es=2; o_result=fail ;;
+
+    esac
+
+    # run preupg
+    #
+    PREUPG__VALID_ES=$o_es \
+        preupg__run1 --skip-common
+    preupg__assert result "$o_result"
+}
+
+xcase__diag() {
+    jat_dump__file module.mdef "$T_MSCRIPT"
+    preupg__get_risks | jat_dump__pipe RESULT_RISKS
+    preupg__get_node | jat_dump__pipe RESULT_NODE
+}
+
+xcase__cleanup() {
+    jat__cmd rm -rf RHEL6_7
+    preupg__rmresult
+}
+
+
+# ............ #                                                               #
+# MAIN() STUFF # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #
+# '''''''''''' #                                                               #
+
+PREUPG__UPATH=@pass \
+    preupg__Run1
+xcase__run
+preupg__Cleanup

src/api/nativity/main.fmf

@@ -0,0 +1,2 @@
+description: package "nativity" functions
+timeout: 3h

src/api/nativity/test

@@ -0,0 +1,263 @@
+#!/bin/bash
+# vim: dict+=/usr/share/beakerlib/dictionary.vim cpt=.,w,b,u,t,i,k
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+#   runtest.sh of /CoreOS/preupgrade-assistant/api/nativity
+#   Description: package "nativity" functions
+#   Author: Alois Mahdal <amahdal@redhat.com>
+#
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#
+#   Copyright (c) 2016 Red Hat, Inc.
+#
+#   This program is free software: you can redistribute it and/or
+#   modify it under the terms of the GNU General Public License as
+#   published by the Free Software Foundation, either version 2 of
+#   the License, or (at your option) any later version.
+#
+#   This program is distributed in the hope that it will be
+#   useful, but WITHOUT ANY WARRANTY; without even the implied
+#   warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+#   PURPOSE.  See the GNU General Public License for more details.
+#
+#   You should have received a copy of the GNU General Public License
+#   along with this program. If not, see http://www.gnu.org/licenses/.
+#
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+shellfu import jat_dump
+shellfu import preupg
+shellfu import preupg_fupath
+shellfu import xcase
+
+#shellcheck disable=SC2034
+{
+    PREUPG__RULE=xccdf_preupg_rule_api_nativity_check
+    JAT_DUMP__LIMIT_L=2000
+    JAT_DUMP__LIMIT_B=20000
+    Fn=
+    Dno=
+    Check=
+    Lang=
+}
+
+EXCLUDED=git
+FOREIGN=preupgrade-assistant-CoreOS-preupgrade-assistant-Library-dummyreq
+NATIVE=rpm
+NINST=habakukova_asymetricka-skladanka
+
+assert_foreign() {
+    #
+    # Assert that package $1 was not deemed native
+    #
+    local pkg=$1
+    case $Fn in
+        get_dist_native_list)
+            rlRun "grep '$pkg' r_native" \
+                1 "package $pkg not considered native"
+            ;;
+        is_dist_native)
+            rlAssertNotEquals \
+                "package $pkg not considered native" \
+                "$(cat r_es)" 0
+            ;;
+    esac
+}
+
+assert_native() {
+    #
+    # Assert that package $1 was deemed native
+    #
+    local pkg=$1
+    case $Fn in
+        get_dist_native_list)
+            rlRun "grep '$pkg' r_native" \
+                0 "package $pkg considered native"
+            ;;
+        is_dist_native)
+            rlAssertEquals \
+                "package $pkg considered native" \
+                "$(cat r_es)" 0
+            ;;
+    esac
+}
+
+xcase__enum() {
+    #
+    # Enumerate subtest ids
+    #
+    # Format:
+    #
+    #     Fn=FN,Dno=DNO,Check=CHECK
+    #
+    # FN is name of relevant function that is tested.
+    #
+    # DNO stands for 'dist_native override', and governs value
+    # passed to preupg__run1() to override
+    # 'native' mechanism,  In this case, it can be:
+    #
+    #  *  'all', 'sign' or empty - pass directly
+    #
+    #  *  'excluded' pass special value to exclude one single package
+    #     from nativity
+    #
+    # CHECK can be (some make sense only for certain functions):
+    #
+    #  *  'foreign' - query for surely non-native package
+    #
+    #  *  'ninst' - query for package that's not installed
+    #
+    #  *  'native' - query for "surely" native package (this can be hard to
+    #     maintain when distribution this test is runnning is under
+    #     development)
+    #
+    #  *  'excluded' - query for package excluded from native by DNO
+    #
+    #  *  'diff' - use standard diff tool to compare expected result to
+    #     actual result
+    #
+    {
+        echo Fn=is_dist_native,Dno=,Check=foreign
+        echo Fn=is_dist_native,Dno=,Check=native
+        echo Fn=is_dist_native,Dno=,Check=ninst
+        echo Fn=get_dist_native_list,Dno=,Check=foreign
+        echo Fn=get_dist_native_list,Dno=,Check=native
+        echo Fn=get_dist_native_list,Dno=all,Check=diff
+        echo Fn=get_dist_native_list,Dno=excluded,Check=diff
+        echo Fn=get_dist_native_list,Dno=excluded,Check=excluded
+#       echo Fn=get_dist_native_list,Dno=sign,Check=diff   #FIXME find out how to get oracle
+#       echo Fn=get_dist_native_list,Dno=,Check=diff       #FIXME find out how to get oracle
+    } | xcase__per Lang py sh
+}
+
+xcase__setup() {
+    #
+    # Prepare for subtest
+    #
+    # Dropped files:
+    #
+    #  *  r_native -- result: native package name list from get_dist_native_list()
+    #  *  r_es     -- result: exit status after calling is_dist_native()
+    #
+    local t_dno     # dist_native override
+
+    # Create module
+    #
+    {
+        echo "[MODULE]"
+        echo "GROUP = api"
+        echo "NAME = nativity"
+        echo "LANG = $Lang"
+        #shellcheck disable=SC2028
+        case $Lang in
+            sh)
+                case $Fn:$Check in
+                    is_dist_native:foreign) echo "CODE = is_dist_native '$FOREIGN'; echo \$? > '$PWD/r_es'" ;;
+                    is_dist_native:native)  echo "CODE = is_dist_native '$NATIVE';  echo \$? > '$PWD/r_es'" ;;
+                    is_dist_native:ninst)   echo "CODE = is_dist_native '$NINST';   echo \$? > '$PWD/r_es'" ;;
+                    get_dist_native_list:*) echo "CODE = get_dist_native_list | LC_ALL=C sort > '$PWD/r_native'" ;;
+                esac
+                echo "CODE = exit_pass"
+                ;;
+            py)
+                echo "CODE = def keep_native(lst):"
+                echo "CODE =     with open('$PWD/r_native', 'w') as fh:"
+                echo "CODE =         fh.write('\\n'.join(sorted(lst)) + '\\n')"
+                echo "CODE = "
+                echo "CODE = def keep_es(bit):"
+                echo "CODE =     with open('$PWD/r_es', 'w') as fh:"
+                echo "CODE =         fh.write('0\\n' if bit else '1\\n')"
+                echo "CODE = "
+                case $Fn:$Check in
+                    is_dist_native:foreign) echo "CODE = keep_es(is_dist_native('$FOREIGN'))" ;;
+                    is_dist_native:native)  echo "CODE = keep_es(is_dist_native('$NATIVE'))"  ;;
+                    is_dist_native:ninst)   echo "CODE = keep_es(is_dist_native('$NINST'))"   ;;
+                    get_dist_native_list:*) echo "CODE = keep_native(get_dist_native_list())" ;;
+                esac
+                echo "CODE = exit_pass()"
+                ;;
+        esac
+    } > module.ini
+    preupg_fupath RHEL6_7 module.ini
+
+    # prepare dist_native override
+    #
+    case $Dno in
+        all)        t_dno="all"         ;;
+        excluded)   t_dno="-$EXCLUDED"  ;;
+        sign)       t_dno="sign"        ;;
+        "")         t_dno=""            ;;
+    esac
+
+    # ensure package (non-)presence)
+    #
+    case $Check in
+        foreign)    rlAssertRpm "$FOREIGN"  ;;
+        native)     rlAssertRpm "$NATIVE"   ;;
+        ninst)      rlAssertNotRpm "$NINST" ;;
+    esac
+
+    # collect data
+    #
+    PREUPG__DISTNATIVE="$t_dno" \
+    PREUPG__UPATH="RHEL6_7/all-xccdf.xml" \
+        preupg__run1
+}
+
+xcase__test() {
+    #
+    # Do the work
+    #
+    # Dropped files:
+    #
+    #  *  rpm_qa   -- all installed package names
+    #  *  o_native -- oracle: all native package names
+    #  *  r_diff   -- result: diff of above two (dumped as error)
+    #
+
+    # prepare oracles
+    #
+    rpm -qa --qf='%{NAME}\n' | LC_ALL=C sort > rpm_qa
+    case $Dno:$Check in
+        all:diff)       cp rpm_qa o_native              ;;
+        excluded:diff)  <rpm_qa grep -vxF "$EXCLUDED" > o_native  ;;
+        sign:diff)      echo 42 >o_native               ;; #FIXME: who knows?
+        :diff)          echo 42 >o_native               ;; #FIXME: who knows?
+    esac
+
+    # do the test
+    #
+    case $Check in
+        diff)       rlRun "diff -U 1 o_native r_native > r_diff" \
+                        0 "native list is same as expected" \
+                     || jat_dump__file -E r_diff ;;
+        native)     assert_native  "$NATIVE"     ;;
+        excluded)   assert_foreign "$EXCLUDED"   ;;
+        foreign)    assert_foreign "$FOREIGN"    ;;
+        ninst)      assert_foreign "$NINST"      ;;
+    esac
+}
+
+xcase__diag() {
+    #
+    # Burp up some diag
+    #
+    jat_dump__file module.ini
+    preupg__get_messages | jat_dump__pipe MESSAGE
+}
+
+xcase__cleanup() {
+    #
+    # Clean up after subtest
+    #
+    preupg__rmresult
+}
+
+
+jat__log_info "EXCLUDED='$EXCLUDED'"
+jat__log_info "FOREIGN='$FOREIGN'"
+jat__log_info "NATIVE='$NATIVE'"
+jat__log_info "NINST='$NINST'"
+
+xcase__run -v
+preupg__Cleanup

src/preupg/devel_mode/main.fmf

@@ -0,0 +1,3 @@
+description: devel mode
+timeout: 30m
+reference: https://bugzilla.redhat.com/show_bug.cgi?id=1231410

src/preupg/devel_mode/test

@@ -0,0 +1,63 @@
+#!/bin/bash
+
+shellfu import jat_dump
+shellfu import preupg
+shellfu import preupg_fupath
+shellfu import xcase
+
+PREUPG__RULE=xccdf_preupg_rule_output_set_check
+PREUPG__DISTNATIVE=""
+
+assert_devel_mode_on() {
+    jat__cmd -h "devel mode is on" \
+        grep 'DEVEL_MODE=1' set.out
+}
+
+assert_devel_mode_not_on() {
+    jat__cmd -h "devel mode is not on" -S 1 \
+        grep 'DEVEL_MODE=1' set.out
+}
+
+
+xcase__enum() {
+    echo on
+    echo off
+}
+
+xcase__setup() {
+    jat__cmd mkdir -p /var/cache/preupgrade
+    case $(xcase__id) in
+        on)     jat__cmd touch /var/cache/preupgrade/devel_mode  ;;
+        off)    jat__cmd rm -f /var/cache/preupgrade/devel_mode  ;;
+    esac
+    {
+        echo '[MODULE]'
+        echo "GROUP = output"
+        echo "NAME = set"
+        echo "CODE = set > '$PWD/set.out'"
+        echo "CODE = exit_pass"
+    } > module.ini
+    preupg_fupath RHEL6_7 module.ini
+    PREUPG__UPATH="RHEL6_7/all-xccdf.xml" \
+        preupg__run1
+}
+
+xcase__test() {
+    case $(xcase__id) in
+        on)     assert_devel_mode_on     ;;
+        off)    assert_devel_mode_not_on ;;
+    esac
+}
+
+xcase__diag() {
+    preupg__get_node | jat_dump__pipe NODE
+    jat_dump__file -l 1000 -b 20000 set.out
+}
+
+xcase__cleanup() {
+    jat__cmd rm -rf /var/cache/preupgrade
+    preupg__rmresult
+}
+
+xcase__run
+preupg__Cleanup

src/preupg/list-rules/main.fmf

@@ -0,0 +1,2 @@
+description: --list-rules argument
+timeout: 90m

src/preupg/list-rules/test

@@ -0,0 +1,120 @@
+#!/bin/bash
+#shellcheck disable=SC1091,SC2034
+
+Installed=
+Custom=
+Select=
+
+shellfu import jat_dump
+shellfu import preupg
+shellfu import preupg_fupath
+shellfu import xcase
+
+xcase__enum() {
+    #
+    # Enumerate use case ids
+    #
+    xcase__permute Installed 0 1 2 \
+      | xcase__per Custom 0 1 2 \
+      | xcase__per Select none s_ok c_ok s_bad c_bad both
+}
+
+mk_cupath() {
+    #
+    # Build custom upath
+    #
+    local name
+    for name in "$@"; do
+        preupg_fupath "$name" @pass @informational @fixed
+    done
+}
+
+xcase__setup() {
+    #
+    # Setup and collect data
+    #
+    local t_selargs      # selection arguments
+
+    case $Installed in
+        0)  :   ;;
+        1)  mk_cupath INS1;     cp -r INS1 /usr/share/preupgrade ;;
+        2)  mk_cupath INS{1,2}; cp -r INS{1,2} /usr/share/preupgrade ;;
+        *)  xcase__id_error Installed ;;
+    esac
+
+    case $Custom in
+        0)  :   ;;
+        1)  mk_cupath CUS1     ;;
+        2)  mk_cupath CUS{1,2} ;;
+        *)  xcase__id_error Custom ;;
+    esac
+
+    case $Select in
+        none)   t_selargs=() ;;
+        s_ok)   t_selargs=(-s INS1) ;;
+        s_bad)  t_selargs=(-s BLAH) ;;
+        c_ok)   t_selargs=(-c CUS1/all-xccdf.xml) ;;
+        c_bad)  t_selargs=(-c CUS3/all-xccdf.xml) ;;
+        both)   t_selargs=(-s INS1 -c CUS1/all-xccdf.xml) ;;
+        *) xcase__id_error Select ;;
+    esac
+
+    jat__cmd -S 0,1,20,22 -o preupg.out -e preupg.err \
+        preupg --list-rules "${t_selargs[@]}"
+        echo $? >r_es
+}
+
+xcase__test() {
+    #
+    # Verify result = oracle
+    #
+    local o_err=false       # oracle: error output?
+    local o_es              # oracle: preupg exit status
+    local o_lines           # oracle: normal output line count
+    local r_err=false       # result: error output?
+    local r_es              # result: preupg exit status
+    local r_lines           # result: normal output line count
+    grep '^xccdf' preupg.out >preupg.out.filtered;  # FIXME: remove and use normal .out
+    r_lines=$(<preupg.out.filtered wc -l)           # after https://github.com/upgrades-migrations/preupgrade-assistant/issues/333
+    test -s preupg.err && r_err=true
+    r_es=$(<r_es)
+
+    case $Installed:$Custom:$Select in
+
+        *:both)     o_lines=0;  o_es=22; o_err=true  ;;
+
+        0:0:*)      o_lines=0;  o_es=20; o_err=true  ;;
+        1:*:none)   o_lines=3;  o_es=0;  o_err=false ;;
+        *:none)     o_lines=0;  o_es=20; o_err=true  ;;
+
+        0:*:s_ok)   o_lines=0;  o_es=20; o_err=true  ;;
+        *:s_ok)     o_lines=3;  o_es=0;  o_err=false ;;
+        *:s_bad)    o_lines=0;  o_es=20; o_err=true  ;;
+
+        *:0:c_ok)   o_lines=0;  o_es=20; o_err=true  ;;
+        *:c_ok)     o_lines=3;  o_es=0;  o_err=false ;;
+        *:c_bad)    o_lines=0;  o_es=20; o_err=true  ;;
+
+        *)  xcase__id_error Installed Custom Select ;;
+    esac
+
+    jat__cmp -h "number of listed lines is correct" \
+        "$r_lines" eq "$o_lines"
+    jat__cmp -h "exit status from preupg is correct" \
+        "$r_es" eq "$o_es"
+#   jat__cmp -h "error output does (not) exist as expected" \
+#       "$r_err" eq "$o_err"
+    # FIXME: enable stderr assert after https://github.com/upgrades-migrations/preupgrade-assistant/issues/333
+}
+
+xcase__diag() {
+    dump__file preupg.out preupg.out.filtered preupg.err
+}
+
+xcase__cleanup() {
+    jat__cmd rm -rf /usr/share/preupgrade/INS*
+    preupg__rmresult
+}
+
+xcase__run -v
+preupg__Cleanup

src/preupg/result-files/main.fmf

@@ -0,0 +1,2 @@
+description: result*.html and similar files
+timeout: 30m

src/preupg/result-files/test

@@ -0,0 +1,163 @@
+#!/bin/bash
+
+shellfu import jat_dump
+shellfu import preupg
+shellfu import preupg_fupath
+
+assert_html_ok() {
+    #
+    # Do sanity test for result.html
+    #
+    local fpath="$1"
+    jat__cmd -h "it's not empty: $fpath" \
+        test -s "$fpath"
+    jat__cmd -h "no _SOLUTION_MSG* placeholders: $fpath" \
+        -S 1 \
+        grep _SOLUTION_MSG "$fpath"
+}
+
+assert_xml_ok() {
+    #
+    # Do sanity test for result.html
+    #
+    local fpath="$1"
+    jat__cmd -h "its not empty: $fpath" \
+        test -s "$fpath"
+    jat__cmd -h "it's a valid XML: $fpath" \
+        xmllint "$fpath"
+}
+
+assert_tarball_ok() {
+    #
+    # Do sanity test for tarball
+    #
+    local fpath="$1"    # path to tarball
+    local fname         # tarball filename
+    local path          # one of mandatory paths from enum_tarball_paths()
+    local pfx           # volatile path prefix (also part of tarball name)
+    fname=$(basename "$fpath")
+    pfx=${fname%.tar.gz}
+    jat__cmd -h "file name determined" \
+        test -n "$fpath"                    || return 1
+    jat__cmd -h "file is not empty" \
+        test -s "$fpath"                    || return 1
+    preupg__tar_list > tarball_contents.txt
+    jat__cmd -h "file contents are not empty" \
+        test -s tarball_contents.txt        || return 1
+    jat__cmd -h "file layout is nested" -S 1 \
+        grep -v "^$pfx/" tarball_contents.txt
+    jat__cmd -h "shadow and gshadow files are not packed" -S 1 \
+        grep -we 'shadow$' -e 'gshadow$' tarball_contents.txt
+    for path in $(enum_tarball_paths);
+    do
+        jat__cmd -h "file $path is present" \
+            grep -qxF "$pfx/$path" tarball_contents.txt
+    done
+    for path in $(enum_tarball_nopaths);
+    do
+        jat__cmd -h "file $path is not present" -S 1 \
+            grep -qxF "$pfx/$path" tarball_contents.txt
+    done
+    jat_dump__file tarball_contents.txt
+}
+
+enum_tarball_paths() {
+    #
+    # Enumerate relative paths that must exist in tarball
+    #
+    # Paths here are listed *without* the "preupg_results-160608163404"
+    # prefix as that is volatile but expected.
+    #
+    echo README
+    echo cleanconf/
+    echo common/
+    echo dirtyconf/
+    echo kickstart/
+    echo noauto_postupgrade.d/
+    echo postupgrade.d/
+    echo preupgrade-scripts/
+    echo result.html
+    echo result.xml
+}
+
+enum_tarball_nopaths() {
+    #
+    # Enumerate relative paths that must NOT exist in tarball
+    #
+    # Paths here are listed *without* the "preupg_results-160608163404"
+    # prefix as that is volatile but expected.
+    #
+    echo result-admin.html  # BZ#1414322
+    echo result-admin.xml   # BZ#1414322
+}
+
+add_3pmod() {
+    #
+    # Add 3rdparty module
+    #
+    # 3rdparty modules need to be added *after* composing, so in order
+    # to take advantage of preupgrade-assistant/fupath, we'll build
+    # separate throw-away set and just move the source files.
+    #
+    preupg_fupath \
+        THIRD \
+        @3rdparty
+    mkdir -p FOO/3rdparty/a_custom_module
+    {
+        echo '[preupgrade]'
+        echo 'group_title = A nice group title'
+    } >FOO/3rdparty/group.ini
+    mv THIRD/3rdparty/a_custom_module/{solution.txt,check,module.ini} \
+        FOO/3rdparty/a_custom_module
+}
+
+jat__pstarts "prepare fake upgrade path"
+    preupg_fupath \
+        FOO \
+        @failed \
+        @fixed \
+        @informational \
+        @needs_action \
+        @needs_inspection \
+        @not_applicable \
+        @pass
+    add_3pmod
+    jat__cmd -h "make backup of upgrade path" \
+        cp -ar FOO FOO-backup
+jat__pend
+
+jat__pstarts "prepare results"
+    PREUPG__UPATH="FOO/all-xccdf.xml" \
+        preupg__run1
+    jat__cmd -h "tarball found" \
+        test -n "$(preupg__get_latest_tar)"
+    preupg__lsrules_ast \
+      | jat_dump__pipe AST_RULES
+jat__pend
+
+jat__pstartt "check if upgrade path has been changed - BZ#1368823"
+    jat__cmd -h "custom upgrade path has not been changed (bz1368823)" \
+        -o updiff \
+        diff -ru FOO FOO-backup
+    jat_dump__file -E updiff
+    test -s updiff && jat__submit updiff FOO.diff
+jat__pend
+
+jat__pstartt "check result*.html"
+    assert_html_ok /root/preupgrade/result.html
+jat__pend
+
+jat__pstartt "check result*.xml"
+    assert_xml_ok /root/preupgrade/result.xml
+jat__pend
+
+jat__pstartt "check tarball"
+    assert_tarball_ok "$(preupg__get_latest_tar)"
+jat__pend
+
+jat__pstartt "check that 3rdparty module has run"
+    preupg__assert -r xccdf_preupg_rule_3rdparty_a_custom_module_check \
+        result pass
+jat__pend
+
+preupg__Cleanup

src/preupg/riskcheck/main.fmf

@@ -0,0 +1,2 @@
+description: riskcheck mode
+timeout: 120m

src/preupg/riskcheck/test

@@ -0,0 +1,86 @@
+#!/bin/bash
+# shellcheck disable=SC2154
+
+shellfu import jat
+shellfu import jat_dump
+shellfu import preupg
+shellfu import xcase
+
+xcase__enum() {
+
+    echo mresult=error,verbose=on
+    echo mresult=failed,verbose=on
+    echo mresult=fixed,verbose=on
+    echo mresult=informational,verbose=on
+    echo mresult=needs_action,verbose=on
+    echo mresult=needs_inspection,verbose=on
+    echo mresult=not_applicable,verbose=on
+    echo mresult=pass,verbose=on
+
+    echo mresult=error,verbose=off
+    echo mresult=failed,verbose=off
+    echo mresult=fixed,verbose=off
+    echo mresult=informational,verbose=off
+    echo mresult=needs_action,verbose=off
+    echo mresult=needs_inspection,verbose=off
+    echo mresult=not_applicable,verbose=off
+    echo mresult=pass,verbose=off
+}
+
+
+xcase__setup() {
+    PREUPG__UPATH="@$mresult" \
+        preupg__run1 --skip-common; echo $?>o_paes
+}
+
+
+xcase__test() {
+    local t_vrbarg=''   # test: verbose argument
+    local o_es=0        # oracle: exit status
+    local o_out=false   # oracle: any stdout?
+    local o_err=false   # oracle: any stderr?
+    local r_es=0        # result: exit status
+    local o_paes        # oracle: preupg exit status
+    o_paes=$(<o_paes)
+
+    case $mresult:$verbose in
+        failed:on)   o_out=true; t_vrbarg="--verbose" ;;
+    esac
+    case $mresult in
+        error)            o_es=2 ;;
+        failed)           o_es=2 ;;
+        fixed)            o_es=0 ;;
+        informational)    o_es=0 ;;
+        needs_action)     o_es=1 ;;
+        needs_inspection) o_es=0 ;;
+        not_applicable)   o_es=0 ;;
+        pass)             o_es=0 ;;
+    esac
+
+    jat__cmd -o preupg.out -e preupg.err -S $o_es \
+        preupg --riskcheck $t_vrbarg; r_es=$?
+    preupg__LibraryLimited loscap "HTML generation warning removed" \
+     && jat__eval "sed -i '/Generating simply styled report due to the limitations of the installed OpenSCAP/d' preupg.out"
+    #FIXME: the message should be in stderr after https://github.com/upgrades-migrations/preupgrade-assistant/issues/315
+    jat__cmp -h "exit status corresponds to module result" \
+        "$r_es" eq "$o_es"
+    jat__cmp -h "exit status is the same as of preupg" \
+        "$r_es" eq "$o_paes"
+    $o_out && jat__cmd -S 0 -h "there is some stdout" test -s preupg.out
+    $o_out || jat__cmd -S 1 -h "there is no stdout"   test -s preupg.out
+    $o_err && jat__cmd -S 0 -h "there is some stderr" test -s preupg.err
+    $o_err || jat__cmd -S 1 -h "there is no stderr"   test -s preupg.err
+}
+
+xcase__diag() {
+    jat_dump__file preupg.out preupg.err
+}
+
+xcase__cleanup() {
+    preupg__rmresult
+}
+
+PREUPG__UPATH=@pass \
+    preupg__Run1
+xcase__run -v
+preupg__Cleanup

src/preupg/select-rules/main.fmf

@@ -0,0 +1,2 @@
+description: --select-rules argument
+timeout: 1h

src/preupg/select-rules/test

@@ -0,0 +1,67 @@
+#!/bin/bash
+
+shellfu import jat
+shellfu import jat_dump
+shellfu import preupg
+shellfu import preupg_fupath
+shellfu import xcase
+
+xcase__enum() {
+    #
+    # Enumerate use case ids
+    #
+    echo one
+    echo two
+    echo all
+}
+
+xcase__setup() {
+    #
+    # Setup and collect data
+    #
+    local regex
+    case $(xcase__id) in
+        one)    regex="pass"    ;;
+        two)    regex="needs_"  ;;
+        all)    regex="."       ;;
+        *)      xcase__id_error ;;
+    esac
+    preupg_fupath \
+        RHEL6_7 \
+        @pass \
+        @informational \
+        @fixed \
+        @needs_action \
+        @needs_inspection
+    PREUPG__UPATH="$PWD/RHEL6_7/all-xccdf.xml" \
+    PREUPG__RULE_REGEX="$regex" \
+        preupg__run1
+}
+
+xcase__test() {
+    #
+    # Setup and collect data
+    #
+    local r_rnum    # result: rule number
+    local o_rnum    # oracle: rule number
+    r_rnum=$(preupg__lsrules_ast | wc -l)
+    case $(xcase__id) in
+        one)    o_rnum=1  ;;
+        two)    o_rnum=2  ;;
+        all)    o_rnum=5  ;;
+        *)      xcase__id_error ;;
+    esac
+    jat__cmp -h "number of ran rules is as expected" \
+        "$r_rnum" eq "$o_rnum"
+}
+
+xcase__diag() {
+    preupg__lsrules_ast | jat_dump__pipe LSRULES_AST
+}
+
+xcase__cleanup() {
+    preupg__rmresult
+}
+
+xcase__run -c /root
+preupg__Cleanup

src/tools/mset-new/main.fmf

@@ -0,0 +1,2 @@
+description: Create new module set
+timeout: 1h

src/tools/mset-new/test

@@ -0,0 +1,54 @@
+#!/bin/bash
+
+shellfu import jat_dump
+
+reply_md() {
+    local case=$1
+    case $case in
+        new)
+            echo FOO
+            echo 6
+            echo 7
+            echo hello
+            echo joe
+            echo sh
+            echo just a test module
+            echo something something
+            ;;
+        add)
+            echo FOO
+            echo y
+            echo hello
+            echo jane
+            echo sh
+            echo just another test module
+            echo something something else else
+            ;;
+    esac
+}
+
+jat__pstartt "create module set"
+    jat__eval "reply_md new | preupg-content-creator"
+    jat__eval "reply_md add | preupg-content-creator"
+    jat__eval "echo log_info Hello >> FOO/hello/joe/check"
+    jat__eval "echo exit_pass >> FOO/hello/joe/check"
+    jat__eval "echo log_info Bye >> FOO/hello/jane/check"
+    jat__eval "echo exit_pass >> FOO/hello/jane/check"
+    jat__eval "preupg-xccdf-compose FOO 1>pxc.out 2>pxc.err"
+    jat__eval "mv FOO FOO-raw; mv FOO-results FOO"
+jat__pend
+
+jat__pstartt "check for specific problems"
+    jat__eval -S 1 -h "solution key was not added (PR#328)" -b pa_pr_328 \
+        "find FOO -name module.ini | xargs -L1 grep -w ^solution | grep ."
+    jat__eval -S 1 -h "no warning about solution key (PR#328)" -b pa_pr_328 \
+        "grep -q 'Warning: tags: .solution. are not allowed in config file' pxc.err"
+jat__pend
+
+jat__pstartd "diag"
+    jat_dump__file pxc.out pxc.err
+jat__pend
+
+jat__pstartt "run module set"
+    jat__cmd preupg -c FOO/all-xccdf.xml --force
+jat__pend