Default Command-Line Options
SYNOPSIS
main.exe [--keep-temp|--delete-temp|--delete-temp-if-success]
[--keep-going|-k] [--global-timeout ]
[--test-timeout ] [--cleanup-timeout ]
[--warn-after-timeout ] [--retry ] [--reset-regressions]
[--on-unknown-regression-files ] [--loop] [--loop-count ]
[--resume-file ] [--resume|-r] [(--job-count|-j) ]
[(--test-arg|-a) <=>...] [--seed ]
[--color|--no-color] [--log-timestamp|--no-log-timestamp]
[--log-prefix|--no-log-prefix] [--log-level ]
[--verbose|-v|--quiet|-q|--info|-i] [--log-file ]
[--log-buffer-size ] [--log-worker-id] [--commands|-c] [--time]
[--record ] [--from-record ...] [--junit ]
[--list|-l|--list-tsv|--suggest-jobs|--version] [(--file|-f) ...]
[--not-file ...] [(--match|-m) ...]
[--not-match ...] [(--title|--test|-t) ...]
[(--not-title|--not-test) ...] [--job </>]
[--skip ] [--only ] [--on-empty-test-list ]
[...]
DESCRIPTION
Run or manage test suite.
Running without any argument runs the full test suite. To see the list of
tests instead of running it, run with the argument `--list`. See below for
more details on how to control test execution and select tests.
OPTIONS
--cleanup-timeout SECONDS (default: 60.)
Send SIGKILL to processes that are still running SECONDS after they were
asked to stop. This affects processes stopped with Process.terminate,
and workers when running with --job-count.
--global-timeout SECONDS
Fail if the set of tests takes more than SECONDS to run.
--job-count COUNT, -j COUNT (default: 1)
Run COUNT tests in parallel, in separate processes. With --suggest-jobs,
set the number of target jobs for --suggest-jobs instead.
If environment variable TEZT_JOB_COUNT is a positive integer, default
value is the value of TEZT_JOB_COUNT. Else, default value is 1.
--keep-going, -k
If a test fails, continue with the remaining tests instead of stopping.
Aborting manually with Ctrl+C still stops everything.
--keep-temp, --delete-temp, --delete-temp-if-success
(default: --delete-temp)
Whether to delete temporary files and directories that were created.
--keep-temp: Keep temporary files and directories after tests.
--delete-temp: Delete temporary files and directories after tests.
--delete-temp-if-success: Delete temporary files and directories, except
if the test failed. If a test succeeds after it however, it causes all
temporary files to be deleted, even those of the failed tests; so you
should probably avoid using this argument in conjunction with
--keep-going or --job-count.
--loop
Restart from the beginning once all tests are done. All tests are
repeated until one of them fails or if you interrupt with Ctrl+C. This
is useful to reproduce non-deterministic failures. When used in
conjunction with --keep-going, tests are repeated even if they fail,
until you interrupt them with Ctrl+C.
--loop-count COUNT
Stop after all tests have been run COUNT times. Implies --loop. A value
of 0 means tests are not run.
--on-unknown-regression-files MODE (default: warn)
How to handle regression test outputs that are not declared by any test.
MODE can be:
- warn: emit a warning for unknown output files;
- ignore: ignore unknown output files;
- fail: terminate execution with exit code 1 and without running any
further action when unknown output files are found;
- delete: delete unknown output files.
To check which files would be deleted, run with this option set to
'warn', which is the default.
--reset-regressions
Remove regression test outputs if they exist, and regenerate them.
--resume, -r
Resume from a previous run. This reads the resume file located at
--resume-file to resume from it.
If --resume-file is not specified, --resume implies --resume-file
tezt-resume.json. If the resume file does not exist, act as if it was
empty.
Before running a test, it is checked whether this test was already
successfully ran according to the resume file. If it was, the test is
skipped.
When using --loop or --loop-count, the test is skipped as many times as
it was successful according to the resume file.
--resume-file FILE
Record test results to FILE for use with --resume. When using --resume,
test results that existed in FILE are kept, contrary to --record.
--retry COUNT (default: 0)
Retry each failing test up to COUNT times. If one retry is successful,
the test is considered successful.
--seed SEED
Force tests declared with ~seed:Random to initialize the pseudo-random
number generator with this seed.
--test-arg =, -a =
Pass a generic argument to tests. Tests can get this argument with
Cli.get. --test-arg is a short-hand for: --test-arg
=true
--test-timeout SECONDS
Fail if a test takes, on its own, more than SECONDS to run.
--warn-after-timeout SECONDS (default: 60.)
If a test is still running SECONDS after it started, warn. This warning
repeats every SECONDS as long as the test is running. If SECONDS is
negative or null, disable this warning instead. This option only has an
effect when using --job-count (-j) with a value of 2 or more.
LOGS
--color, --no-color (default: --no-color)
Whether to use colors in output. Default value depends on whether stdout
is a terminal and on the value of the TERM environment variable.
--commands, -c
Output commands which are run, in a way that is easily copy-pasted for
manual reproducibility.
--log-buffer-size COUNT (default: 50)
Before logging an error on stdout, also log the last COUNT messages that
have been ignored because of the log level since the last message that
was not ignored.
--log-file FILE
Also log to FILE. Note that --log-level does not apply: FILE contains
logs in verbose mode. In the presence of --job-count, the main process
will log test results to FILE while each worker writes test logs to a
separate file BASENAME-WORKER_ID[.EXT]. BASENAME is the basename of
FILE, WORKER_ID is the zero-indexed id of the worker and .EXT is the
extension of FILE if present.
--log-level LEVEL (default: report)
Set log level. Possible LEVELs are: quiet, error, warn, report, info,
debug.
--log-prefix, --no-log-prefix (default: --log-prefix)
Whether to print log prefixes, i.e. the '~prefix' argument of log
functions.
--log-timestamp, --no-log-timestamp (default: --log-timestamp)
Whether to print a timestamp.
--log-worker-id
Decorate logs with worker IDs when --job-count is more than 1.
--verbose, -v, --quiet, -q, --info, -i
Set log level. Overrides --log-level.
--verbose, -v: Same as --log-level debug.
--quiet, -q: Same as --log-level quiet.
--info, -i: Same as --log-level info.
REPORTS
--from-record FILE
Use data recorded with --record. If specified multiple times, use the
union of those records.
If is a directory, this is equivalent to specifying --from-record
for all files in this directory that have the .json extension.
When using --time, test durations include tests found in record files.
When using --record, the new record which is output does NOT include the
input records.
When using --junit, reports do NOT include input records.
--junit FILE
Store test results in FILE using JUnit XML format. Time information for
each test is the sum of all runs of this test for the current session.
Test result (success or failure) is the result for the last run of the
test.
--record FILE
Record test results to FILE. This file can then be used with
--from-record. If you use --loop or --loop-count, times are averaged for
each test.
--time
Print a summary of the total time taken by each test. Ignored if a test
failed. Includes the time read from records: to display a record, you
can use --from-record --list --time.
COMMANDS
--list, -l, --list-tsv, --suggest-jobs, --version
Do not run tests. Instead:
--list, -l: List tests.
Pass --time to also display results and timings (in seconds) from a
previous execution given through --from-record, in the format TIME
(COUNT). TIME is the average time of successful executions. COUNT is
SCOUNT/(SCOUNT+FCOUNT) where SCOUNT (resp. FCOUNT) is the number of
successful (resp. failed) tests in the record. If there is only one
successful test, then (COUNT) is omitted. Tests lacking a past record of
successful executions are noted '-'. A final row is added containing the
total of the averages of successful test executions, and the total
number of selected tests.
--list-tsv: List tests as tab-separated values in the format FILE TITLE
TAGS.
Pass --time to also display results and timings (in nanoseconds) from a
previous execution given through --from-record. Then each line is
appended with STIME SCOUNT FTIME FCOUNT. STIME (resp. FTIME) is the
total running time in nanoseconds of successful (resp. failed) previous
runs. SCOUNT (resp. FCOUNT) is the count of successful (resp. failed)
previous runs.
--suggest-jobs: Read test results records specified with --from-record
and suggest a partition of the tests that would result in --job-count
sets of roughly the same total duration. Output each job as a list of
flags that can be passed to Tezt, followed by a shell comment that
denotes the expected duration of the job.
A similar result can be obtained with --list --job, except that the last
job suggested by --suggest-jobs uses --not-test to express "all tests
that are not already in other jobs", meaning that the last job acts as a
catch-all for unknown tests.
--version: Print the version number of Tezt and exit.
SELECTING TESTS
Tests are registered with a filename, a title, and a list of tags. You can
specify multiple tags, negated tags, titles, title patterns and filenames on
the command line. Only tests which match all the following conditions will
be run:
- the test must have all tags and none of the negated tags;
- the test must have one of the specified titles;
- the test must have a title matching one of the specified patterns;
- the test must be implemented in one of the specified files.
The tags of a test are given by the ~tags argument of Test.register. To
negate a tag, prefix it with a slash: /
The title of a test is given by the ~title argument of Test.register. It is
what is printed after [SUCCESS] (or [FAILURE] or [ABORTED]) in the reports.
Use --title (respectively --not-title) to select (respectively unselect) a
test by its title on the command-line. You can also select (respectively
unselect) tests for which 'filename: title' matches one or several Perl
regular expressions using --match (respectively --not-match).
The file in which a test is implemented is specified by the ~__FILE__
argument of Test.register. In other words, it is the path of the file in
which the test is defined. Use --file (respectively --not-file) to select
(respectively unselect) a test by its path (or a suffix thereof) on the
command-line.
For instance:
_build/default/test/unix/main.exe node bake /rpc --file bootstrap.ml --file
sync.ml
will run all tests defined in either bootstrap.ml or sync.ml, which have at
least tags 'node' and 'bake', but which do not have the 'rpc' tag.
You can also specify more complex predicates using the Test Selection
Language (TSL). See section TEST SELECTION LANGUAGE (TSL) below.
--file FILE, -f FILE
Only run tests implemented in source files ending with FILE.
--job /
Split the set of selected tests into COUNT subsets of roughly the same
total duration. Execute only one of these subsets, specified by INDEX.
COUNT must be at least 1 and INDEX must be between 1 and COUNT.
Use --from-record to feed duration data from past runs. Tests for which
no time data is available are given a default duration of 1 second.
You can use --list to see what tests are in a subset without actually
running the tests.
A typical use is to run tests in parallel on different machines. For
instance, have one machine run with --job 1/3, one with --job 2/3 and
one with --job 3/3. Be sure to provide exactly the same records with
--from-record, in the same order, and to select exactly the same set of
tests (same tags, same --file and same --test) for all machines,
otherwise some tests may not be run at all.
--match PERL_REGEXP, -m PERL_REGEXP
Only run tests for which 'FILE: TITLE' matches PERL_REGEXP (case
insensitive), where FILE is the source file of the test and TITLE its
title.
--not-file FILE
Only run tests not implemented in source files ending with FILE.
--not-match PERL_REGEXP
Only run tests for which 'FILE: TITLE' does not match PERL_REGEXP (case
insensitive), where FILE is the source file of the test and TITLE its
title.
--not-title VALUE, --not-test VALUE
Only run tests which are not exactly entitled TITLE.
--on-empty-test-list MODE (default: fail)
How to behave if the list of selected tests is empty. MODE can be:
- ignore: exit with code 0;
- warn: output 'No test found for filters: ...', exit with code 0;
- fail: output 'No test found for filters: ...', exit with code 3.
--only COUNT
Only run the first COUNT tests. This filter is applied after --job and
--skip.
--skip COUNT (default: 0)
Skip the first COUNT tests. This filter is applied after --job and
before --only.
--title VALUE, --test VALUE, -t VALUE
Only run tests which are exactly entitled TITLE.
TSL_EXPRESSION
Only run tests that satisfy the predicate denoted by TSL_EXPRESSION.
TEST SELECTION LANGUAGE (TSL)
For more advanced needs, you can select tests using Test Selection Language
(TSL) expressions. For instance:
_build/default/test/unix/main.exe 'file = bootstrap.ml || bake && /rpc'
runs all tests defined in bootstrap.ml, as well as tests (possibly from
other files) with tag 'bake' but without tag 'rpc'. This example cannot be
expressed without TSL.
Passing multiple predicates is equivalent to passing their conjunction as a
single predicate. For instance, passing the two command-line arguments 'a ||
b' and 'c || d' is equivalent to passing the single argument '(a || b) && (c
|| d)'.
TSL allows the following expressions, where STRING denotes a string and EXPR
denotes a TSL expression:
- 'true': always true;
- 'false': always false;
- 'EXPR && EXPR': conjunction;
- 'EXPR || EXPR': disjunction;
- 'not EXPR': negation;
- 'STRING': test has tag STRING;
- '/STRING': same as 'not STRING' (STRING must not be quoted);
- 'file = STRING': test file is STRING;
- 'file <> STRING': same as 'not (file = STRING)';
- 'title = STRING': test title is STRING;
- 'title <> STRING': same as 'not (title = STRING)';
- 'file =~ STRING': test file matches the regular expression STRING;
- 'file =~! STRING': same as 'not (file =~ STRING)';
- 'title =~ STRING': test title matches the regular expression STRING;
- 'title =~! STRING': same as 'not (title =~ STRING)';
- '(EXPR)': same as EXPR.
'not' has higher precedence than '&&' which has higher precedence than '||'.
TSL strings need to be quoted using double quotes '"' unless they only
contain characters 'a-zA-Z0-9_-./' and do not start with a slash '/'. Double
quotes '"' and backslashes '\' need to be escaped using backslashes '\'.
Note that 'file = STRING' is not equivalent to '--file STRING'. Indeed,
'--file a.ml' selects 'x/a.ml', while 'file = a.ml' does not.
Examples of TSL expressions:
'/tag && title = test_title'
'not (some_tag || "some_other_tag") && title <> "test title"'
'file =~ "some *regular *expression"'