jats-preupgrade-assistant

preupgrade-assistant JAT test suite

This is an independent test suite for preupgrade-assistant that makes use of JATS - Just A Test System.

JATS defines test suite format, which is a directory tree consisting of individual tests. These individual tests can in principle be implemented in any language, but for now shellfu-bash-jat is the only available library.

Dependencies - test machine

All specific dependencies are available in COPR projects netvor/jats and netvor/shellfu. However, since preupgrade-assistant itself is only available for RHEL6, which does not have native COPR support, you will need to add repositories manually.

On the testing machine, you will need:

• shellfu-bash-jat - the main JAT library,
• shellfu-bash-preupg - preupgrade-assistant test library,
• jattool - harness, exploration and reporting toolkit
• jats-preupgrade-assistant - this test suite

Note that tests don't formally depend on preupgrade-assistant itself, so you will need to install it manually. (This is to help you retain full control of the preupgrade-assistant version; in future there will be mechanisms to help with that but for now manual is the safest.).

Also note that you don't need to install all sub-components of jattool: jattool-tdk and jattool-report require things not available easily on all RHEL versions and are normally only useful on workstations anyway.

Last but not least: you don't really need to install the jats-preupgrade-assistant; you could do away with just cloning this repo. However, RPM will help you with the dependencies, so it's advised to install it even if you want to run from the repo.

Running tests

At this point, most tests should work without installation. That is, just clone this repo and run

jattool tfind

to see lists of tests. Each JAT test is a directory, so above should give you list of directories: you can then run any of them using jattool runtest, eg.:

jattool runtest ./src/api/binary_req

However, you can also install tests, which ensures that tests are versioned properly (and version logged within test is correct).

yum install jats-preupgrade-assistant
jattool tfind /usr/share
jattool runtest /usr/share/jats/suite/preupgrade-assistant/preupg/result-files

should give you proper version:

test.INFO: id: jats://redhat.com/oamg/preupgrade-assistant//preupg/result-files
test.INFO: version: 0.0.0

session.START

phase.START 'prepare fake upgrade path'
  test.INFO: CMD: rm -rf FOO FOO-raw FOO-results
[...]

Dependencies - workstation

Since your workstation is probably Fedora, you can use the modern ways:

dnf copr enable netvor/shellfu
dnf copr enable netvor/jats
dnf install shellfu-bash-jat jattool jattool-report jattool-tdk
            #\
    # don't be put off of the number of dependencies; let's just say
    # that author of Shellfu loves granularity :-)

This will give you basic "test development kit". More on that later.

Exploring (and hacking on) tests

I won't go into details; I'll tell you just enough to get you started, but be aware that (mostly) everything is (mostly) documented and primary ways to get help (actually more like reference, but helps) using sfdoc tool, which should be on your workstation by now.

Tests normally only consist of one important file, named ... you guessed it... test. (There's also main.fmf, but that only contains meta-data, which is not even really used these days.)

$ tree src/api/binary_req
src/api/binary_req
├── main.fmf
└── test

So open that file: it's a Bash script with extra "enhancements": you'll see things like:

shellfu import jat
shellfu import preupg
shellfu import ...
shellfu import xcase

Shellfu platform brings modularity to Bash, so the names above are names of modules. jat is a module name. And you can get reference on the module

sfdoc jat

Since this is Bash, modules contain just functions and variables. And since Bash has no namespaces, an important convention for (almost) all modules is in place:

• any functions defined by module foo must be named by foo__ prefix,
• any variables defined by module foo must be named by FOO__ prefix.

Eg. an initializing function from module bar would have to be called bar__init().

Yep, this is a bit annoying, but:

• You'll get used to it.

• Since Bash has no namespaces, this is the only way to avoid conflicts without having to read every imported module's source code (recursively, and after every update).

• It can actually be helpful: seeing function like jat__eval, you immediately know it's from jat module (heck, you can call sfdoc -O jat__eval).

• The above also applies to possible tools you might want to develop (it's possible to parse module name from object name).

• It provides nice excuse to keep module and function names short!

Creating new tests

NOTE: At the time of this writing, the test creation kit is not finished, so if you really need new test, you can just copy another one.

In order to avoid duplication of work and unnecessary frustration, take following steps:

1. Before you start, contact maintainer and inform them that you are planning to write a test. You can do this by filing an issue in pagure.io repo or by contacting the maintainer directly.

2. Read Shellfu coding style; tests need to follow this.

3. Be aware that the JATS project is in early stage (v0.0.*), and some (most?) pieces of coding style and conventions are not yet written down. A more consistent documentation is expected to arrive before v0.1.0.