Introduction¶
Atheist is a simple framework for running test cases. You write small files in Python language using a set of predefined functions and classes. In many senses, the concept is quite similar to make or the SCons framework although Atheist is not a building system at all.
Features:
- Black-box testing: system and acceptance tests. It may execute any kind of external shell program.
- White-box testing by means of unitest and standard Python functions.
- Python doctests.
- Python unittest.
- New kinds of “tests” by means of plugins.
- Plugable pre- and post- condition system.
ToDo:
- Per-project plugins.
- Limit/perf testing.
- Remote testing.
- Test deployment.
- Remote management.
- Distributed testing.
Test objects¶
The Test object is the minimal testing unit. Each Test instance
defines an individual execution (a shell command) that may be
checked for success upon termination. The Test constructor accepts many
parameters that may change the test’s exception behavior in several
ways. The only mandatory parameter is cmd
which is the command to
execute.
The Test object is responsible for executing the command and checking its termination value. A very basic example:
Test('true')
Test files (TestCases)¶
Test instances need to be defined in text source files (with
.test
extension). Although these files are written in a subset of
the Python language, they may be seen as declarative programs. You tell
Atheist what you want to test and even the order, but the decision
about when to run the corresponding action is taken by Atheist; some
of them may be never done.
The file does not define sequential imperative sentences. For example,
if you write this in a .test
file:
Test('ls /')
print "hello"
the print
statement will be executed when the test is LOADED
and the ls
command will run later, when the test is
EXECUTED. You must take in mind that the atheist .test file defines
a set of tests. It is not a conventional python program.
Key-args¶
Any Test constructor accepts the next key-val parameters. All of them are optional. Beside the parameter’s name appear its type and default value.
check – type: bool
, default: True
If ‘check’ isFalse
there is no consequences when the Task fails and it is not considered in the final stats.
cwd – type: str
Directory for the task execution.
delay – type: int
, default: 0
Waits for ‘delay’ seconds before launching the task actions.
desc – type: str
One-liner textual task description.
detach – type: bool
, default: False
Whendetach
isFalse
the next task does not start until the current one ends. Whendetach
isTrue
the next task is executed even if the current one is still running.
env – type: str
:str
map
A dictionary with shell environment variables.
expected – type: int
Expected return code for the command. It may be negative if the process is killed by a signal.
tid – type: str
It is a unique string Task IDentifier. You can get a reference to the task later by giving this value to theget_task()
function.
must_fail – type: bool
, default: False
When you expect the program to end with an error but the return code is not known (i.e: is different from zero). You should check other conditions (stdout, stderr, generated files, etc) to differentiate alternative fails.
parent – type: CompositeTask
Use this to aggregate the current test to an already defined CompositeTask.
template – type: Template list
A list of templates. See Templates.
timeout – type: int, default: 5
The maximum task execution time (in seconds). When the timeout finishes, Atheist sends the programmed signal to the process. To avoid timed termination (daemonic task) givetimeout=0
.
save_stderr – type: bool
, default: False
Store the process’ stderr in a text file. If thestderr
parameter is not set, Atheist will create an unique name for the file.
save_stdout – type: bool
, default: False
Store the process’ stdout in a text file. If thestdout
parameter is not set, Atheist will create an unique name for the file.
shell – type: bool
, default: False
Execute the command within a shell session.bash
is the used shell.
signal – type: int, default: SIGKILL
It is the signal that Atheist sends to the process when thetimeout
finishes.
stdout – type: str
It is the name of the file where to save the process’ stdout. Setting this parameters implies save_stdout = True.
todo – type: bool
, default: False
It indicates that the task is not fully verified and it is possible that it fail unduly. This has no effect when the task ends successfully.
Not all of these key-args are available for all Task classes. See Task’s, Test’s, Commands, Daemons….
Task results¶
The execution of any task returns a value, which can be:
- FAIL: The task ran normally but user requirements or conditions were not met. The test failed.
- OK: The task ran successfully and all required conditions and/or return values were correct.
- NOEXEC: The task was skipped or it was not executed.
- ERROR: The implementation of the task is wrong and the task execution failed itself.
- UNKNOWN: The task was executed but its result is not known.
- TODO: The task implementation is unstable and it may produce false failures.
Templates¶
The template is a set of predefined values for Test key-values. You may use the same configuration for many tests avoiding to repeat them. This is an example:
t1 = Template(timeout=6, expected=-9)
Test('foo', templates=[t1])
Test('bar', templates=[t1])
Both tests will be automatically killed after 6 seconds and the expected return value is -9. This means that these processes receive the SIGKILL(9) signal. You may specify several templates as a list.
Conditions¶
Conditions are predicates (actually, functors) that check for specific conditions. Conditions may be specified to be checked before (pre-conditions) or after (post-conditions) the task execution. If any of the conditions fail, then the task fails. This is an example:
t = Test('foo')
t.pre += FileExists('/path/to/foofile')
t.post += FileContains('path/to/barfile', 'some text')
Available builtin conditions¶
-
AtheistVersion
(version)¶ Checks that the installed version of atheist is equal or newer than the given number. This is useful to assure recent or experimental features.
-
Callback
(*args)¶ Call the specified function with the given arguments. You must avoid the use of this condition as much as possible.
-
DirExists
(path)¶ Checks that directory path exists.
-
EnvVarDefined
(name[, value])¶ Checks that the environment variable name exists, and optionally has the value value.
-
FileExists
(filename)¶ Checks that file filename exists.
-
FileContains
(val[, filename=task.stdout, strip='', whole=False, times=1])¶ Checks that file filename exists and contains val, which may be a string or a string list.
If whole is
False
(default) val must be a strip and the whole content of file must be equal to val.Otherwise, the file must contain at least times occurrences of val. The default value for filename is the stdout of the corresponding task, and it implies
save_stdout=True
. This implies also the automatic creation of a FileExists condition for that file.The file content may be stripped by means of strip argument. No stripping by default.
-
FileEquals
(filename1[, filename2=task.stdout])¶ Checks that the contents of filename1 and filename2 are identical. The default value for filename2 is the stdout of the current task, and it implies
save_stdout=True
.
-
OpenPort
(port[, host='localhost'[, proto='tcp']])¶ Checks that port number port is open, i.e., a process is listening to it.
-
ProcessRunning
(pid)¶ Checks that the given PID belongs to a running process.
There are other available conditions as plugins.
Condition decorators¶
-
Not
(condition)¶ It is True when condition is evaluated as False.
Example:
t = Test('rm foo_file') t.pre += Not(FileExists(foo_file))
-
Poll
(condition[, interval=1[, timeout=5]])¶ Checks condition every interval seconds, stopping after timeout seconds or when its value becomes True.
In the next example, the task waits (a maximum of 5 seconds) for the
nc
server to become ready before continuing:t = Test('nc -l -p 2000') t.post += Poll(OpenPort(2000))
Condition decorators may be combined. The following example shows a task that waits for an environment variable to be removed before executing the command:
t = Test('command')
t.pre += Poll(Not(EnvVarDefined('WAIT_FLAG')), timeout=10)
Note that the effect of Poll(Not(condition)) is not the same that Not(Poll(condition)).
Task’s, Test’s, Commands, Daemons…¶
Task
is the base class for all executable items. Test
is-a
Task
that runs a shell command but other kind of Task
are
possible:
- Command
It is a non-checked
Test
. Command is exactly the same that a Test with acheck=False
parameter.The Commands (or other non-checked Tasks) are not considered in results counting.
- Daemon
Command shortcut for detached commands. Predefined parameters are:
detach = True
expected = -9
(sigkilled)timeout = None
(runs “forerver”)check = False
- TestFunc
- Check the return value of arbitrary Python functions or methods. A return value of 0 means success, otherwise is a error code. For unit testing, prefer `unittestcase`_ instead of TestFunc.
There are other available Task subclasses as plugins.
Function utilities¶
-
get_task
(name)¶ Returns the task whose
tid
attribute is name. ToDo: [include a sample here]
-
load
(filename)¶ Makes possible to reuse atheist or python code in other files.
load()
returns a module-like object that may be used to access to functions, classes and variables defined in the “loaded” module. All atheist classes are available within the loaded modules:common = load("files/common.py") Test("./server %s" % common.server_params())
Warning
Previous include()
function is not supported any more.
Variable substitutions¶
Test files may include some substitutable variable. This is useful to locate test-relevant related files. You must write the symbol ‘$’ preceding each one of the next words:
- basedir
- is the name of the directory where atheist was
executed. Usually this is a
tests
directory into your project. - dirname
- is the name of the directory where the testfile is.
- fname
- is the path to the testfile without its extension (.test).
- testname
- is just the name of the testfile, without extension nor directory path.
For example, for the following vars.test
file:
Test("echo $basedir $dirname $fname $testname")
When you run atheist, you get:
~/sample$ atheist -i2 tests/vars.test
[ OK ] Test case: ./tests/substitutions.test
[ OK ] +- T-1 ( 0: 0) echo . ./tests ./tests/vars vars
Total: ALL OK!! - 0.24s - 1 test
hooks: setup and teardown¶
You may write tasks to execute before and after each test file. To do
so, just put this tasks in files called _setup.test
and
_teardown.test
.
Clean-up¶
When your task creates files you may track them for automatic
cleaning. Just add their filenames to the task gen
attribute. Here’s an
example:
t = Test('touch foo')
t.gen += 'foo'
You may specify one o more filenames (as a string list).
If you want the generated files not to be automatically removed for
manual inspection of the results, you must specify the --dirty
option
(see below). To clean-up these files later, specify the -C
option.
Invoking Atheist¶
-
-h
,
--help
¶
Show basic help information.
-
-a
ARGS
,
--task-args
=ARGS
¶ Colon-separated options for the tests.
-
-b
PATH
,
--base-dir
=PATH
¶ Change working directory.
-
-C
,
--clean-only
¶
Don’t execute anything, only remove generated files.
-
-d
,
--describe
¶
Describe tasks but don’t execute anything.
-
-e
,
--stderr
¶
Print the test process’ stderr.
-
-f
,
--out-on-fail
¶
Print task output, but only if it fails.
-
-g
,
--gen-template
¶
Generate a test file template with default values.
-
-i
LEVEL
,
--report-detail
=LEVEL
¶ Verbosity level (0:nothing, [1:case], 2:task, 3:composite, 4:condition)
-
-j
,
--skip-hooks
¶
Skip _setup and _teardown files.
-
-k
,
--keep-going
¶
Continue despite failed tests.
-
-l
,
--list
¶
List tests but do not execute them.
-
-o
,
--stdout
¶
Print the test process’ stdout.
-
-p
PATH
,
--plugin-dir
=PATH
¶ Load plugins from that directory (may be given several times).
Print the test process’ stdout.
-
-q
,
--quiet
¶
Do not show result summary nor warnings, only totals.
-
-r
RANDOM
,
--random
=RANDOM
¶ Run testcases in random order using the specified seed.
-
-s
INLINE
,
--script
=INLINE
¶ Specifies command line script.
-
-t
,
--time-tag
¶
Include time info in the logs.
-
-u
,
--until-failure
¶
Repeat tests until something fails.
-
-v
,
--verbose
¶
Increase verbosity.
-
-w
WORKERS
,
--workers
=WORKERS
¶ Number of simultaneous tasks. ‘0’ allows atheist to choose the number. Default is 1.
-
-x
COMMAND
,
--exec
=COMMAND
¶ Exec COMMAND like in “Test(COMMAND, shell=True)”
-
--case-time
¶
Print case execution time in reports
-
--cols
=WIDTH
¶ Set terminal width (in chars).
-
--config
=FILE
¶ Alternate config file.
-
--dirty
¶
Don’t remove generated files after test execution.
-
--disable-bar
¶
Don’t show progress bar.
-
--ignore
=PATTERN
¶ Files to ignore (glob patterns) separated with semicolon.
-
--log
=LOG
¶ List tasks but do not execute them.
-
--plain
¶
Avoid color codes in console output.
-
--save-stdout
¶
Save stdout of all tasks
-
--save-stderr
¶
Save stderr of all tasks
-
--version
¶
Atheist version
-
--notify-jabber
=JABBER
¶ Notify failed tests to the given jabber account (may be given several times).
-
--notify-smtp
=MAIL
¶ Notify failed tests to the given email address (may be given several times).
Logging control¶
[ToDo]
Result report¶
[ToDo]
Config files¶
[ToDo]
Notifications (other reporters)¶
You may use Atheist to monitor the proper working of any application. It
may send you notifications when something is wrong. The easiest way is to
run a testcase with cron
specifying one --notify-*
command-line
argument. Currently, two notificators are implemented:
The destination is a email account using the SMTP protocol. Atheist does not require an SMTP server or smarthost. You only need to configure an email account that will be used by Atheist to send mail to any destination. That information must be written in the
~/.atheist/config
configuration file. This is an example:[smtp] host = smtp.server.org port = 587 user = atheist.notify@server.org pasw = somesecret
- Jabber
You must indicate a jabber account that will be used by Atheist to send notification messages to any other jabber account. The destination account must accept this contact previously. The following is an example for the configuration in the
~/.atheist/config
file:[jabber] user = atheist.notify@server.org pasw = anothersecret
To ask for a notification you just need to specify a test file and one or more destinations. For example:
$ atheist --notify-jabber John.Doe@jabber.info test/some_test.test
It is possible to give several –notify-* arguments in the same command-line to send notifications to different destinations.