Tezt API

Default Command-Line Options

SYNOPSIS

    main.exe [--keep-temp|--delete-temp|--delete-temp-if-success]
        [--keep-going|-k] [--global-timeout ]
        [--test-timeout ] [--cleanup-timeout ]
        [--warn-after-timeout ] [--retry ] [--reset-regressions]
        [--on-unknown-regression-files ] [--loop] [--loop-count ]
        [--resume-file ] [--resume|-r] [(--job-count|-j) ]
        [(--test-arg|-a) <=>...] [--seed ]
        [--color|--no-color] [--log-timestamp|--no-log-timestamp]
        [--log-prefix|--no-log-prefix] [--log-level ]
        [--verbose|-v|--quiet|-q|--info|-i] [--log-file ]
        [--log-buffer-size ] [--log-worker-id] [--commands|-c] [--time]
        [--record ] [--from-record ...] [--junit ]
        [--list|-l|--list-tsv|--suggest-jobs|--version] [(--file|-f) ...]
        [--not-file ...] [(--match|-m) ...]
        [--not-match ...] [(--title|--test|-t) ...]
        [(--not-title|--not-test) ...] [--job </>]
        [--skip ] [--only ] [--on-empty-test-list ]
        [...]

DESCRIPTION

    Run or manage test suite.

    Running without any argument runs the full test suite. To see the list of
    tests instead of running it, run with the argument `--list`. See below for
    more details on how to control test execution and select tests.

OPTIONS

    --cleanup-timeout SECONDS (default: 60.)
        If a process spawned by a test is still running SECONDS after it was
        sent SIGTERM at the end of the test, send it SIGKILL.

    --global-timeout SECONDS
        Fail if the set of tests takes more than SECONDS to run.

    --job-count COUNT, -j COUNT (default: 1)
        Run COUNT tests in parallel, in separate processes. With --suggest-jobs,
        set the number of target jobs for --suggest-jobs instead.

        If environment variable TEZT_JOB_COUNT is a positive integer, default
        value is the value of TEZT_JOB_COUNT. Else, default value is 1.

    --keep-going, -k
        If a test fails, continue with the remaining tests instead of stopping.
        Aborting manually with Ctrl+C still stops everything.

    --keep-temp, --delete-temp, --delete-temp-if-success
    (default: --delete-temp)
        Whether to delete temporary files and directories that were created.

        --keep-temp: Keep temporary files and directories after tests.

        --delete-temp: Delete temporary files and directories after tests.

        --delete-temp-if-success: Delete temporary files and directories, except
        if the test failed. If a test succeeds after it however, it causes all
        temporary files to be deleted, even those of the failed tests; so you
        should probably avoid using this argument in conjunction with
        --keep-going or --job-count.

    --loop
        Restart from the beginning once all tests are done. All tests are
        repeated until one of them fails or if you interrupt with Ctrl+C. This
        is useful to reproduce non-deterministic failures. When used in
        conjunction with --keep-going, tests are repeated even if they fail,
        until you interrupt them with Ctrl+C.

    --loop-count COUNT
        Stop after all tests have been run COUNT times. Implies --loop. A value
        of 0 means tests are not run.

    --on-unknown-regression-files MODE (default: warn)
        How to handle regression test outputs that are not declared by any test.
        MODE can be:
        - warn: emit a warning for unknown output files;
        - ignore: ignore unknown output files;
        - fail: terminate execution with exit code 1 and without running any
        further action when unknown output files are found;
        - delete: delete unknown output files.

        To check which files would be deleted, run with this option set to
        'warn', which is the default.

    --reset-regressions
        Remove regression test outputs if they exist, and regenerate them.

    --resume, -r
        Resume from a previous run. This reads the resume file located at
        --resume-file to resume from it.

        If --resume-file is not specified, --resume implies --resume-file
        tezt-resume.json. If the resume file does not exist, act as if it was
        empty.

        Before running a test, it is checked whether this test was already
        successfully ran according to the resume file. If it was, the test is
        skipped.

        When using --loop or --loop-count, the test is skipped as many times as
        it was successful according to the resume file.

    --resume-file FILE
        Record test results to FILE for use with --resume. When using --resume,
        test results that existed in FILE are kept, contrary to --record.

    --retry COUNT (default: 0)
        Retry each failing test up to COUNT times. If one retry is successful,
        the test is considered successful.

    --seed SEED
        Force tests declared with ~seed:Random to initialize the pseudo-random
        number generator with this seed.

    --test-arg =, -a =
        Pass a generic argument to tests. Tests can get this argument with
        Cli.get. --test-arg  is a short-hand for: --test-arg
        =true

    --test-timeout SECONDS
        Fail if a test takes, on its own, more than SECONDS to run.

    --warn-after-timeout SECONDS (default: 60.)
        If a test is still running SECONDS after it started, warn. This warning
        repeats every SECONDS as long as the test is running. If SECONDS is
        negative or null, disable this warning instead. This option only has an
        effect when using --job-count (-j) with a value of 2 or more.

LOGS

    --color, --no-color (default: --no-color)
        Whether to use colors in output. Default value depends on whether stdout
        is a terminal and on the value of the TERM environment variable.

    --commands, -c
        Output commands which are run, in a way that is easily copy-pasted for
        manual reproducibility.

    --log-buffer-size COUNT (default: 50)
        Before logging an error on stdout, also log the last COUNT messages that
        have been ignored because of the log level since the last message that
        was not ignored.

    --log-file FILE
        Also log to FILE. Note that --log-level does not apply: FILE contains
        logs in verbose mode. In the presence of --job-count, the main process
        will log test results to FILE while each worker writes test logs to a
        separate file BASENAME-WORKER_ID[.EXT]. BASENAME is the basename of
        FILE, WORKER_ID is the zero-indexed id of the worker and .EXT is the
        extension of FILE if present.

    --log-level LEVEL (default: report)
        Set log level. Possible LEVELs are: quiet, error, warn, report, info,
        debug.

    --log-prefix, --no-log-prefix (default: --log-prefix)
        Whether to print log prefixes, i.e. the '~prefix' argument of log
        functions.

    --log-timestamp, --no-log-timestamp (default: --log-timestamp)
        Whether to print a timestamp.

    --log-worker-id
        Decorate logs with worker IDs when --job-count is more than 1.

    --verbose, -v, --quiet, -q, --info, -i
        Set log level. Overrides --log-level.

        --verbose, -v: Same as --log-level debug.
        --quiet, -q: Same as --log-level quiet.
        --info, -i: Same as --log-level info.

REPORTS

    --from-record FILE
        Use data recorded with --record. If specified multiple times, use the
        union of those records.

        If  is a directory, this is equivalent to specifying --from-record
        for all files in this directory that have the .json extension.

        When using --time, test durations include tests found in record files.

        When using --record, the new record which is output does NOT include the
        input records.

        When using --junit, reports do NOT include input records.

    --junit FILE
        Store test results in FILE using JUnit XML format. Time information for
        each test is the sum of all runs of this test for the current session.
        Test result (success or failure) is the result for the last run of the
        test.

    --record FILE
        Record test results to FILE. This file can then be used with
        --from-record. If you use --loop or --loop-count, times are averaged for
        each test.

    --time
        Print a summary of the total time taken by each test. Ignored if a test
        failed. Includes the time read from records: to display a record, you
        can use --from-record  --list --time.

COMMANDS

    --list, -l, --list-tsv, --suggest-jobs, --version
        Do not run tests. Instead:

        --list, -l: List tests.

        Pass --time to also display results and timings (in seconds) from a
        previous execution given through --from-record, in the format TIME
        (COUNT). TIME is the average time of successful executions. COUNT is
        SCOUNT/(SCOUNT+FCOUNT) where SCOUNT (resp. FCOUNT) is the number of
        successful (resp. failed) tests in the record. If there is only one
        successful test, then (COUNT) is omitted. Tests lacking a past record of
        successful executions are noted '-'. A final row is added containing the
        total of the averages of successful test executions, and the total
        number of selected tests.

        --list-tsv: List tests as tab-separated values in the format FILE TITLE
        TAGS.

        Pass --time to also display results and timings (in nanoseconds) from a
        previous execution given through --from-record. Then each line is
        appended with STIME SCOUNT FTIME FCOUNT. STIME (resp. FTIME) is the
        total running time in nanoseconds of successful (resp. failed) previous
        runs. SCOUNT (resp. FCOUNT) is the count of successful (resp. failed)
        previous runs.

        --suggest-jobs: Read test results records specified with --from-record
        and suggest a partition of the tests that would result in --job-count
        sets of roughly the same total duration. Output each job as a list of
        flags that can be passed to Tezt, followed by a shell comment that
        denotes the expected duration of the job.

        A similar result can be obtained with --list --job, except that the last
        job suggested by --suggest-jobs uses --not-test to express "all tests
        that are not already in other jobs", meaning that the last job acts as a
        catch-all for unknown tests.

        --version: Print the version number of Tezt and exit.

SELECTING TESTS

    Tests are registered with a filename, a title, and a list of tags. You can
    specify multiple tags, negated tags, titles, title patterns and filenames on
    the command line. Only tests which match all the following conditions will
    be run:
    - the test must have all tags and none of the negated tags;
    - the test must have one of the specified titles;
    - the test must have a title matching one of the specified patterns;
    - the test must be implemented in one of the specified files.

    The tags of a test are given by the ~tags argument of Test.register. To
    negate a tag, prefix it with a slash: /

    The title of a test is given by the ~title argument of Test.register. It is
    what is printed after [SUCCESS] (or [FAILURE] or [ABORTED]) in the reports.
    Use --title (respectively --not-title) to select (respectively unselect) a
    test by its title on the command-line. You can also select (respectively
    unselect) tests for which 'filename: title' matches one or several Perl
    regular expressions using --match (respectively --not-match).

    The file in which a test is implemented is specified by the ~__FILE__
    argument of Test.register. In other words, it is the path of the file in
    which the test is defined. Use --file (respectively --not-file) to select
    (respectively unselect) a test by its path (or a suffix thereof) on the
    command-line.

    For instance:

    _build/default/test/unix/main.exe node bake /rpc --file bootstrap.ml --file
    sync.ml

    will run all tests defined in either bootstrap.ml or sync.ml, which have at
    least tags 'node' and 'bake', but which do not have the 'rpc' tag.

    You can also specify more complex predicates using the Test Selection
    Language (TSL). See section TEST SELECTION LANGUAGE (TSL) below.

    --file FILE, -f FILE
        Only run tests implemented in source files ending with FILE.

    --job /
        Split the set of selected tests into COUNT subsets of roughly the same
        total duration. Execute only one of these subsets, specified by INDEX.

        COUNT must be at least 1 and INDEX must be between 1 and COUNT.

        Use --from-record to feed duration data from past runs. Tests for which
        no time data is available are given a default duration of 1 second.

        You can use --list to see what tests are in a subset without actually
        running the tests.

        A typical use is to run tests in parallel on different machines. For
        instance, have one machine run with --job 1/3, one with --job 2/3 and
        one with --job 3/3. Be sure to provide exactly the same records with
        --from-record, in the same order, and to select exactly the same set of
        tests (same tags, same --file and same --test) for all machines,
        otherwise some tests may not be run at all.

    --match PERL_REGEXP, -m PERL_REGEXP
        Only run tests for which 'FILE: TITLE' matches PERL_REGEXP (case
        insensitive), where FILE is the source file of the test and TITLE its
        title.

    --not-file FILE
        Only run tests not implemented in source files ending with FILE.

    --not-match PERL_REGEXP
        Only run tests for which 'FILE: TITLE' does not match PERL_REGEXP (case
        insensitive), where FILE is the source file of the test and TITLE its
        title.

    --not-title VALUE, --not-test VALUE
        Only run tests which are not exactly entitled TITLE.

    --on-empty-test-list MODE (default: fail)
        How to behave if the list of selected tests is empty. MODE can be:
        - ignore: exit with code 0;
        - warn: output 'No test found for filters: ...', exit with code 0;
        - fail: output 'No test found for filters: ...', exit with code 3.

    --only COUNT
        Only run the first COUNT tests. This filter is applied after --job and
        --skip.

    --skip COUNT (default: 0)
        Skip the first COUNT tests. This filter is applied after --job and
        before --only.

    --title VALUE, --test VALUE, -t VALUE
        Only run tests which are exactly entitled TITLE.

    TSL_EXPRESSION
        Only run tests that satisfy the predicate denoted by TSL_EXPRESSION.

TEST SELECTION LANGUAGE (TSL)

    For more advanced needs, you can select tests using Test Selection Language
    (TSL) expressions. For instance:

    _build/default/test/unix/main.exe 'file = bootstrap.ml || bake && /rpc'

    runs all tests defined in bootstrap.ml, as well as tests (possibly from
    other files) with tag 'bake' but without tag 'rpc'. This example cannot be
    expressed without TSL.

    Passing multiple predicates is equivalent to passing their conjunction as a
    single predicate. For instance, passing the two command-line arguments 'a ||
    b' and 'c || d' is equivalent to passing the single argument '(a || b) && (c
    || d)'.

    TSL allows the following expressions, where STRING denotes a string and EXPR
    denotes a TSL expression:
    - 'true': always true;
    - 'false': always false;
    - 'EXPR && EXPR': conjunction;
    - 'EXPR || EXPR': disjunction;
    - 'not EXPR': negation;
    - 'STRING': test has tag STRING;
    - '/STRING': same as 'not STRING' (STRING must not be quoted);
    - 'file = STRING': test file is STRING;
    - 'file <> STRING': same as 'not (file = STRING)';
    - 'title = STRING': test title is STRING;
    - 'title <> STRING': same as 'not (title = STRING)';
    - 'file =~ STRING': test file matches the regular expression STRING;
    - 'file =~! STRING': same as 'not (file =~ STRING)';
    - 'title =~ STRING': test title matches the regular expression STRING;
    - 'title =~! STRING': same as 'not (title =~ STRING)';
    - '(EXPR)': same as EXPR.

    'not' has higher precedence than '&&' which has higher precedence than '||'.

    TSL strings need to be quoted using double quotes '"' unless they only
    contain characters 'a-zA-Z0-9_-./' and do not start with a slash '/'. Double
    quotes '"' and backslashes '\' need to be escaped using backslashes '\'.

    Note that 'file = STRING' is not equivalent to '--file STRING'. Indeed,
    '--file a.ml' selects 'x/a.ml', while 'file = a.ml' does not.

    Examples of TSL expressions:

    '/tag && title = test_title'
    'not (some_tag || "some_other_tag") && title <> "test title"'
    'file =~ "some *regular *expression"'