JATS - JAT-based test suite

Version

v0.0 -- proof-of-concept phase

Disclaimer: Version number of this specification starts with zero, which signifies proof-of-concept phase; many aspects including the basic ones WILL CHANGE.

Suite

JATS test suite consists of one or more JAT-based tests. Each test is a directory, containing:

test
meta.yml

test is main body of the test. meta.yml is a YAML file containing test meta-data such as human-readable description, dependencies and coverage information.

Root directory

Root directory of a test suite is designated by placement of file format in sub-directory .jats. This file MUST contain version number in X.Y format, where X is major format version and Y is minor format version.

A tool declaring support for JATS test suite MUST recognize one or more compatible format versions and refuse to work with suite if no supported versions match suite version.

Suite meta-data

Test suite meta-data is stored under .jats directory under respective filename:

domain - Domain under which test suite exists. A public JATS test suite SHOULD refer to a valid domain name owned by test suite owner.

Default value is jats.example.org.
ns - Namespace under which rules for test organization are defined. MUST consist of valid Path as defined in RFC 3986.

Usually this contains hierarchical elements such as work group name and project name.

Default value is anonns.
sut_pname - Plain name (or "package name") of System Under Test. MUST consist of lowercase ASCII letters and dashes. MUST NOT start with dash.

Default value is anonsut.
sut_url - Valid HTTP(s) URL leading to more information about System Under Test. Default value is https://example.org/anonsut,
url - Valid HTTP(s) URL leading to more information about Test Suite itself. Default value is composed in following way:
```
https://<DOMAIN>/<NS>
```
where <DOMAIN> and <NS> are contents of files domain and ns as above, respectively.

Test

By default, the format of the test is a source-able Bash code using shellfu-bash-jat library. A trivial example of test code could be:

jat__pstartt "example test phase"
    jat__cmd -h "true is still true" \
        true
    jat__cmp -h "five is still greater than three" \
        5 gt 3
    jat__cmp -h "foo still contains oo" \
        foo re oo
jat__pend

Meta-data and test properties

Properties of a test are inferred from directory structure and complemented by test meta-data. Following test properties are defined:

Test id

Test id is determined from available data and has format of:

jats://<DOMAIN>/<NS>/<SUT_PNAME>/<PATH>

where:

jats:// is a constant pseudo-protocol string,
<DOMAIN>, <NS>, and <SUT_PNAME> are contents of respective files from .jats meta-data directory,
and <PATH> is path of test directory relative to suite root directory.

For example, following tree:

foosuite
  +-- .jats
  |       +-- domain        # contaning "my.example.org"
  |       +-- ns            # contaning "myns"
  |       +-- sut_pname     # contaning "mysut"
  |       +-- version
  +-- foo
  |     +-- test
  +-- bar
       +-- baz-simple
       |     +-- test
       +-- baz-advanced
             +-- test

contains following tests:

jats://my.example.org/myns/mysut/foo
jats://my.example.org/myns/mysut/bar/baz-simple
jats://my.example.org/myns/mysut/bar/baz-advanced

SUITE.md 3.7KB History Raw