|
@@ -1,360 +0,0 @@
|
1
|
|
-Tests
|
2
|
|
-=====
|
3
|
|
-
|
4
|
|
-Running tests is handled by tfkit/runtests:
|
5
|
|
-
|
6
|
|
- $ tfkit/runtest [filter]
|
7
|
|
-
|
8
|
|
-*filter* is a regular expression to be applied to sub-test name, running
|
9
|
|
-only the matching ones. See below for details.
|
10
|
|
-
|
11
|
|
-
|
12
|
|
-Writing tests
|
13
|
|
--------------
|
14
|
|
-
|
15
|
|
-Tests can be written in any scripting language, although the built-in
|
16
|
|
-framework, written in Bash, provides some useful features for writing
|
17
|
|
-certain kind of relatively simple tests.
|
18
|
|
-
|
19
|
|
-The harness, though, assumes that:
|
20
|
|
-
|
21
|
|
- * Any direct sub-directory of `$TF_SUITE` directory ("tests" by default)
|
22
|
|
- that contains at least *TF_RUN* executable becomes a test,
|
23
|
|
-
|
24
|
|
- * basename of this directory becomes the name of the test,
|
25
|
|
-
|
26
|
|
- * and return code from running the executable is reported
|
27
|
|
- as result of the test, according to "Exit status" chapter below.
|
28
|
|
-
|
29
|
|
-
|
30
|
|
-Naming
|
31
|
|
-------
|
32
|
|
-
|
33
|
|
-Test name should start with name of the module that is tested and
|
34
|
|
-underscore. If module name contains dots, they should be replaced with
|
35
|
|
-underscores as well.
|
36
|
|
-
|
37
|
|
- core_sanity
|
38
|
|
- mod_submod_function
|
39
|
|
- ini_iniread
|
40
|
|
-
|
41
|
|
-are valid test names.
|
42
|
|
-
|
43
|
|
-
|
44
|
|
-Data
|
45
|
|
-----
|
46
|
|
-
|
47
|
|
-Should the test need any data, just leave it around in the test directory
|
48
|
|
-along with *TF_RUN*.
|
49
|
|
-
|
50
|
|
-Note that before running, the whole test directory is automatically
|
51
|
|
-copied to a temporary location (one per test), and should the test fail,
|
52
|
|
-copied back as a debugging artifact. For this reason, *do not store
|
53
|
|
-huge amounts of data here*. If you really need huge data, consider
|
54
|
|
-obtaining it (and throwing it away) within runtime of *TF_RUN*.
|
55
|
|
-
|
56
|
|
-
|
57
|
|
-Exit status
|
58
|
|
------------
|
59
|
|
-
|
60
|
|
-We try hard to follow this semantic:
|
61
|
|
-
|
62
|
|
- * *Zero* means *OK* -- test has been run and passed.
|
63
|
|
-
|
64
|
|
- * *One* means *Failure* -- test has been run but failed (e.g. found
|
65
|
|
- a bug).
|
66
|
|
-
|
67
|
|
- * *Two* means *Bailout* -- test has decided not to run at all.
|
68
|
|
-
|
69
|
|
- * *Three* means *Error* -- there was error detected during execution,
|
70
|
|
- but script was able to clean up properly.
|
71
|
|
-
|
72
|
|
- * *Four* means *Panic* -- there was other error but script *was not*
|
73
|
|
- able to clean up properly.
|
74
|
|
-
|
75
|
|
- * Anything else should indicate other uncaught errors, including those
|
76
|
|
- outside control of the program such as segfaults in the test code
|
77
|
|
- or test being SIGKILLed.
|
78
|
|
-
|
79
|
|
-Notice that the higher the value is, the worse situation it indicates.
|
80
|
|
-Thus, if a test is composed of several sub-tests, you need to make sure
|
81
|
|
-to always **exit with the highest value** (subtest.sh does take care
|
82
|
|
-of this).
|
83
|
|
-
|
84
|
|
-See *common.sh* for functions and variables to help with handling exit
|
85
|
|
-statuses with this semantic.
|
86
|
|
-
|
87
|
|
-Also see Notes section for more details on exit statuses, including
|
88
|
|
-cheat sheet and dscussuion.
|
89
|
|
-
|
90
|
|
-
|
91
|
|
-Framework
|
92
|
|
----------
|
93
|
|
-
|
94
|
|
-
|
95
|
|
-### harness.sh ###
|
96
|
|
-
|
97
|
|
-This part is not intended to be used in tests, but rather contains
|
98
|
|
-functions that help govern test discovery, preparation and execution as
|
99
|
|
-is described in previous chapters. Feel free to poke around, of course.
|
100
|
|
-
|
101
|
|
-
|
102
|
|
-### subtest.sh ###
|
103
|
|
-
|
104
|
|
-As name suggests, this file defines few functions to handle subtests
|
105
|
|
-in *TF_RUN*.
|
106
|
|
-
|
107
|
|
-In order to make use of the subtests functionality, you will need to
|
108
|
|
-define two functions yourself: `tf_enum_subtests` to enumerate names of
|
109
|
|
-tests you want to run, and `tf_name2cmd` to translate each name an actual
|
110
|
|
-command that would perform it and return with the the correct exit status.
|
111
|
|
-
|
112
|
|
-The minimal *TF_RUN* with two subtests could look like this:
|
113
|
|
-
|
114
|
|
- #!/bin/bash
|
115
|
|
-
|
116
|
|
- . $TF_DIR/include/subtest.sh
|
117
|
|
-
|
118
|
|
- tf_enum_subtests() {
|
119
|
|
- echo test1
|
120
|
|
- echo test2
|
121
|
|
- something && echo test3
|
122
|
|
- }
|
123
|
|
-
|
124
|
|
- tf_name2cmd() {
|
125
|
|
- case $1 in
|
126
|
|
- test1) echo myprog foo ;;
|
127
|
|
- test2) echo myprog bar ;;
|
128
|
|
- esac
|
129
|
|
- }
|
130
|
|
-
|
131
|
|
- tf_do_subtests
|
132
|
|
-
|
133
|
|
-At the end, `tf_do_subtests` acts as a launcher of the actual test.
|
134
|
|
-In short, it will
|
135
|
|
-
|
136
|
|
- * take each enumerated subtest from `tf_enum_subtests`,
|
137
|
|
- * source *TF_SETUP*, if such file is found,
|
138
|
|
- * translate te subtest name to a command,
|
139
|
|
- * launch the command,
|
140
|
|
- * source *TF_CLEANUP*, if such file is found,
|
141
|
|
- * and report "worst" exit status encountered.
|
142
|
|
-
|
143
|
|
-All but the first and last step is done by `tf_do_subtest`, so in some
|
144
|
|
-cases you may want to re-define this one as well.
|
145
|
|
-
|
146
|
|
-Note that subtest names need to be single words (`[a-zA-Z0-9_]`).
|
147
|
|
-
|
148
|
|
-
|
149
|
|
-### tools.sh ###
|
150
|
|
-
|
151
|
|
-This file contains various tools and utilities to help with testing.
|
152
|
|
-
|
153
|
|
-Curently there is only one function, `tf_testflt` designed to help write
|
154
|
|
-tests for simple unix filters.
|
155
|
|
-
|
156
|
|
-
|
157
|
|
-#### tf_testflt ####
|
158
|
|
-
|
159
|
|
-The idea is that tester specifies
|
160
|
|
-
|
161
|
|
- * test name,
|
162
|
|
- * command to launch the system under test,
|
163
|
|
- * a data stream to use as STDIN,
|
164
|
|
- * and expected STDOUT, STDERR, and exit status.
|
165
|
|
-
|
166
|
|
-and tf_testflt launches the command, collects tha data and evaluates
|
167
|
|
-and reports the result using unified diff.
|
168
|
|
-
|
169
|
|
-In its simplest form:
|
170
|
|
-
|
171
|
|
- tf_testflt -n foo my_command arg
|
172
|
|
-
|
173
|
|
-the function will run `my_command arg` (not piping anything to it),
|
174
|
|
-and will expect it to finish with exit status 0 and empty both STDERR
|
175
|
|
-and STDOUT.
|
176
|
|
-
|
177
|
|
-Example of full form,
|
178
|
|
-
|
179
|
|
- tf_testflt -n foo -i foo.in -O foo.stdout -E foo.stderr -S 2 myprog
|
180
|
|
-
|
181
|
|
-will pipe foo.in into `myprog`, expecting exit status of 2, and STDOUT and
|
182
|
|
-STDERR as above. Notice that parameters specifying expected values are
|
183
|
|
-uppercase, and those specifying input values are lowercase.
|
184
|
|
-
|
185
|
|
-Specifying name is mandatory, because it's used in reporting messages,
|
186
|
|
-and as a basis for naming temporary result files: these are saved in
|
187
|
|
-*results* subdirectory and kept for further reference.
|
188
|
|
-
|
189
|
|
-
|
190
|
|
-### common.sh ###
|
191
|
|
-
|
192
|
|
-This includes simple functions and variables shared between both mentioned
|
193
|
|
-libraries.
|
194
|
|
-
|
195
|
|
-First group is designed to help support the exit status semantic:
|
196
|
|
-
|
197
|
|
- * The functions are `tf_exit_pass`, `tf_exit_fail`, `tf_exit_bailout`,
|
198
|
|
- `tf_exit_error` and `tf_exit_panic` and each take any number of
|
199
|
|
- parameters that are printed on stderr.
|
200
|
|
-
|
201
|
|
- * The variables are `TF_ES_OK`, `TF_ES_FAIL`, `TF_ES_BAILOUT`,
|
202
|
|
- `TF_ES_ERROR` and `TF_ES_PANIC` and are supposed to be used with
|
203
|
|
- `return` builtin, e.g. to return from `tf_exit_error`.
|
204
|
|
-
|
205
|
|
-Second group is useful to better control output: functions `tf_warn`,
|
206
|
|
-`tf_debug` and `tf_think` are used to print stuff on STDERR. Use of
|
207
|
|
-`tf_warn` is apparent, just as `tf_debug`, the latter being muted if
|
208
|
|
-`TF_DEBUG` is set to `false` (set it to `true` to turn on debugging).
|
209
|
|
-
|
210
|
|
-`tf_think` is used for progress info, and is muted unless `TF_VERBOSE`
|
211
|
|
-is set to `true`, which is by default.
|
212
|
|
-
|
213
|
|
-
|
214
|
|
-### Setup and cleanup ###
|
215
|
|
-
|
216
|
|
-Special files *TF_SETUP* and *TF_CLEANUP* (one of them or both) can be
|
217
|
|
-added along with *TF_RUN*. These will be sourced before (*TF_SETUP*)
|
218
|
|
-and after every subtest (*TF_CLEANUP*).
|
219
|
|
-
|
220
|
|
-First, if any of these files are missing, it is considered as if the
|
221
|
|
-respective phase succeeded. Second, if setup phase fails, test will
|
222
|
|
-be skipped and subtest exit status will be *TF_ES_BAILOUT*. Last,
|
223
|
|
-if cleanup fails (no matter result of setup), subtests aborts with
|
224
|
|
-*TF_ES_PANIC* returned. Be aware that in this case the actual test
|
225
|
|
-status, albeit useful, is lost.
|
226
|
|
-
|
227
|
|
-When coming from other test frameworks, this may feel harsh, but note
|
228
|
|
-that this has been designed with the idea that if a cleanup fails,
|
229
|
|
-it may render all further tests are automatically unsafe, because the
|
230
|
|
-environment is not as expected.
|
231
|
|
-
|
232
|
|
-To cope with this behavior, try to bear in mind following advice:
|
233
|
|
-
|
234
|
|
- 1. Make sure you write setup/cleanup procedures with extreme care and
|
235
|
|
- test them well.
|
236
|
|
-
|
237
|
|
- 2. Do not do complicated and risky things in the setup/cleanup phases.
|
238
|
|
-
|
239
|
|
- 3. If you need to do such things, consider doing them in the *TF_RUN*
|
240
|
|
- instead of doing them for all subtests.
|
241
|
|
-
|
242
|
|
- 4. You don't need to clean up everything, the contents of the testing dir
|
243
|
|
- will be moved out from the test system.
|
244
|
|
-
|
245
|
|
- 5. If there are scenarios you can safely fix or ignore, handle them in
|
246
|
|
- a robust manner.
|
247
|
|
-
|
248
|
|
-
|
249
|
|
-Notes
|
250
|
|
------
|
251
|
|
-
|
252
|
|
-
|
253
|
|
-### bailout vs. `tf_enum_subtests` ###
|
254
|
|
-
|
255
|
|
-One more note to claify relation of bailout and `tf_enum_subtests`.
|
256
|
|
-As you may have noticed, there are two ways how to skip a test:
|
257
|
|
-return prematurely with `TF_ES_BAILOUT`, or suppress enumeration in
|
258
|
|
-`tf_enum_subtests`. The problem is that the latter does not do anything
|
259
|
|
-to inform upper in the stack that a test has been skipped, which seems to
|
260
|
|
-break the principle described in the previous chapters.
|
261
|
|
-
|
262
|
|
-Don't confuse these mechanisms, though. Each is supposed to be used
|
263
|
|
-for distinct purpose. Compare: by using the `tf_enum_subtests` you are
|
264
|
|
-saying that you actually **did not even want** to run the test in the
|
265
|
|
-first place. By using `TF_ES_BAILOUT`, you are saying that you **wanted**
|
266
|
|
-to run the test but could not.
|
267
|
|
-
|
268
|
|
-A few common cases if that helps you:
|
269
|
|
-
|
270
|
|
- * If during the test you find out that for some reason it can't be
|
271
|
|
- carried out (e.g. an external resource is not available, or
|
272
|
|
- something outside the SUT is broken), use `TF_ES_BAILOUT`.
|
273
|
|
-
|
274
|
|
- * If you want to disable the test because for some long-term condition,
|
275
|
|
- e.g. a known bug outside SUT but preventing execution of the test
|
276
|
|
- is not fixed, use `tf_enum_subtests`.
|
277
|
|
-
|
278
|
|
- * If you want to filter out some sub-tests to only for some platforms,
|
279
|
|
- e.g. 64-bit architecture, (IOW, you can safely check that a
|
280
|
|
- sub-test would be totally pointless if run on this box), use
|
281
|
|
- `tf_enum_subtests`.
|
282
|
|
-
|
283
|
|
- * If you want to disable (comment out test) that you might not have
|
284
|
|
- implemented yet or is broken (and for some reason you still want
|
285
|
|
- it to haunt the test code), use `tf_enum_subtests` and properly
|
286
|
|
- comment the reasons in code.
|
287
|
|
-
|
288
|
|
- * If in doubt, use `TF_ES_BAILOUT`.
|
289
|
|
-
|
290
|
|
-
|
291
|
|
-### On exit statuses: three and above ###
|
292
|
|
-
|
293
|
|
-The difference in *error*, *panic* and higher values is subtle but
|
294
|
|
-important. Follow me as I try to explain:
|
295
|
|
-
|
296
|
|
- 1. If script has changed something on the system outside the working
|
297
|
|
- directory, it is apparently expected to revert that change.
|
298
|
|
-
|
299
|
|
- 2. Now if an error occurs, but the code responsible for cleaning up is
|
300
|
|
- safely run, you can say there was *error but we have recovered*.
|
301
|
|
-
|
302
|
|
- 3. But if the change can't be reverted safely, we know that we have
|
303
|
|
- broken something and latter code may lead to weird results (including
|
304
|
|
- masking bugs(!)), it's time to *panic* (in the code, not in real
|
305
|
|
- life ;))
|
306
|
|
-
|
307
|
|
- 4. And then there are corner cases like a bug in the script, OOM kill
|
308
|
|
- or timeout when the status will be different and not really controlled
|
309
|
|
- by the script. Such cases will have to be treated the same way as
|
310
|
|
- the "panic" case, but...
|
311
|
|
-
|
312
|
|
- 5. the use of *panic* adds hint that the status has been set consciously
|
313
|
|
- by the script, albeit exiting "in a hurry"--without proper clean up.
|
314
|
|
-
|
315
|
|
-Unfortunately there will be cases like above but with the error code less
|
316
|
|
-than four. Example is a bash script syntax error, which returns 2, or
|
317
|
|
-Python exception which returns 1. Yes, in such cases the information
|
318
|
|
-conveyed by the exit status is wrong and you should do everything to
|
319
|
|
-avoid it.
|
320
|
|
-
|
321
|
|
-Possibilities like "test has passed but then something blew up" exist,
|
322
|
|
-but conveying this information is responsibility of the test output.
|
323
|
|
-
|
324
|
|
-Following table can be used as a cheat-sheet:
|
325
|
|
-
|
326
|
|
- .---------------------------------------------------------------.
|
327
|
|
- | e | state of | |
|
328
|
|
- | s |---------------------| script says |
|
329
|
|
- | | SUT | environment | |
|
330
|
|
- |---|-------|-------------|-------------------------------------|
|
331
|
|
- | 0 | OK | safe | test passed, everything worked fine |
|
332
|
|
- | 1 | buggy | safe | test failed, everything worked fine |
|
333
|
|
- | 2 | ??? | safe | I decided not to run the test |
|
334
|
|
- | 3 | ??? | safe | Something blew up but I managed to |
|
335
|
|
- | | | | clean up (I promise!) |
|
336
|
|
- | 4 | ??? | broken | Something blew up and I rushed out |
|
337
|
|
- | | | | in panic |
|
338
|
|
- | * | ??? | broken | ...nothing (is dead) |
|
339
|
|
- '---------------------------------------------------------------'
|
340
|
|
-
|
341
|
|
-As you can see, following this semantic allows us to see both the state
|
342
|
|
-of the system under test (SUT) *and* the environment.
|
343
|
|
-
|
344
|
|
-Following table illustrates how different statuses map to different
|
345
|
|
-scenarios with regard to test result as well as state of the environment:
|
346
|
|
-
|
347
|
|
- .--------------------------------------------------.
|
348
|
|
- | environment | test result | test result |
|
349
|
|
- | | pass fail unkn | pass fail unkn |
|
350
|
|
- |-------------|----------------|-------------------|
|
351
|
|
- | clean(ed) | 0 1 3 | OK FAIL ERROR |
|
352
|
|
- | untouched | ~ ~ 2 | ~ ~ BAILOUT |
|
353
|
|
- | mess | ~ ~ 4 | ~ ~ PANIC |
|
354
|
|
- | ?! (trap) | ~ ~ 5 | ~ ~ ~ |
|
355
|
|
- | ?! (sig 9) | ~ ~ 137 | ~ ~ ~ |
|
356
|
|
- | ?! (aliens) | ~ ~ ? | ~ ~ ~ |
|
357
|
|
- '-------------|----------------|-------------------|
|
358
|
|
- | exit status | human-readable |
|
359
|
|
- | | name (TF_ES_*) |
|
360
|
|
- '------------------------------------'
|