preupgrade-assistant JAT test suite
===================================

This is an independent test suite for preupgrade-assistant that makes use
of JATS - Just A Test System.

JATS defines test suite format, which is a directory tree consisting of
individual tests.  These individual tests can in principle be implemented
in any language, but for now *shellfu-bash-jat* is the only available
library.


Dependencies - test machine
---------------------------

All specific dependencies are available in COPR projects [netvor/jats][nj] and
[netvor/shellfu][ns].  However, since preupgrade-assistant itself
is only available for RHEL6, which does not have native COPR support,
you will need to add repositories manually.

  [nj]: https://copr.fedorainfracloud.org/coprs/netvor/jats/
  [ns]: https://copr.fedorainfracloud.org/coprs/netvor/shellfu/

On the testing machine, you will need:

 *  shellfu-bash-jat - the main JAT library,
 *  shellfu-bash-preupg - preupgrade-assistant test library,
 *  jattool - harness, exploration and reporting toolkit
 *  jats-preupgrade-assistant - this test suite

Note that tests don't formally depend on preupgrade-assistant itself, so you will
need to install it manually. (This is to help you retain full control of the
preupgrade-assistant version; in future there will be mechanisms to help with that
but for now manual is the safest.).

Also note that you don't need to install all sub-components of jattool: jattool-tdk
and jattool-report require things not available easily on all RHEL versions and are
normally only useful on workstations anyway.

Last but not least: you don't really need to install the jats-preupgrade-assistant;
you could do away with just cloning this repo.  However, RPM will help you with
the dependencies, so it's advised to install it even if you want to run from the
repo.


Running tests
-------------

At this point, most tests should work without installation.   That is, just
clone this repo and run

    jattool tfind

to see lists of tests.   Each JAT test is a directory, so above should give
you list of directories: you can then run any of them using `jattool runtest`,
eg.:

    jattool runtest ./src/api/binary_req


However, you can also install tests, which ensures that tests are versioned
properly (and version logged within test is correct).

    yum install jats-preupgrade-assistant
    jattool tfind /usr/share
    jattool runtest /usr/share/jats/suite/preupgrade-assistant/preupg/result-files

should give you proper version:

    test.INFO: id: jats://redhat.com/oamg/preupgrade-assistant//preupg/result-files
    test.INFO: version: 0.0.0

    session.START

    phase.START 'prepare fake upgrade path'
      test.INFO: CMD: rm -rf FOO FOO-raw FOO-results
    [...]


Dependencies - workstation
--------------------------

Since your workstation is probably Fedora, you can use the modern ways:

    dnf copr enable netvor/shellfu
    dnf copr enable netvor/jats
    dnf install shellfu-bash-jat jattool jattool-report jattool-tdk
                #\
        # don't be put off of the number of dependencies; let's just say
        # that author of Shellfu loves granularity :-)

This will give you basic "test development kit".  More on that later.


Exploring (and hacking on) tests
--------------------------------

I won't go into details; I'll tell you just enough to get you started,
but be aware that **(mostly) everything is (mostly) documented** and
primary ways to get help (actually more like reference, but helps)
using *sfdoc* tool, which should be on your workstation by now.

Tests normally only consist of one important file, named ... you
guessed it...  *test*.  (There's also *main.fmf*, but that only contains
meta-data, which is not even really used these days.)

    $ tree src/api/binary_req
    src/api/binary_req
    ├── main.fmf
    └── test

So open that file:  it's a Bash script with extra "enhancements":  you'll see things
like:

    shellfu import jat
    shellfu import preupg
    shellfu import ...
    shellfu import xcase

Shellfu platform brings modularity to Bash, so the names above are names
of modules.  `jat` is a module name.  And you can get reference on the module

    sfdoc jat

Since this is Bash, modules contain just functions and variables.  And since
Bash has no namespaces, an important convention for (almost) all modules
is in place:

 *  any functions defined by module `foo` must be named by `foo__` prefix,
 *  any variables defined by module `foo` must be named by `FOO__` prefix.

Eg. an initializing function from module *bar* would have to be called
`bar__init()`.

Yep, this is a bit annoying, but:

 *  You'll get used to it.

 *  Since Bash has no namespaces, this is the only way to avoid conflicts
    without having to read every imported module's source code (recursively,
    and after every update).

 *  It can actually be helpful: seeing function like `jat__eval`, you
    *immediately know* it's from *jat* module (heck, you can call `sfdoc
    -O jat__eval`).

 *  The above also applies to possible tools you might want to develop (it's
    possible to *parse* module name from object name).

 *  It provides nice excuse to keep module and function names short!


Creating new tests
------------------

**NOTE:** At the time of this writing, the test creation kit is not finished,
so if you really need new test, you can just copy another one.

In order to avoid duplication of work and unnecessary frustration,
take following steps:

 1. Before you start, **contact maintainer** and inform them that you
    are planning to write a test.  You can do this by filing an issue
    in pagure.io repo or by contacting the maintainer directly.

 2. Read [Shellfu coding style][sfcs]; tests need to follow this.

 3. Be aware that the JATS project is in early stage (`v0.0.*`), and
    some (most?) pieces of coding style and conventions are not yet
    written down.  A more consistent documentation is expected to arrive
    before `v0.1.0`.

  [sfcs]: https://pagure.io/shellfu/src/last/notes/style.md