diff options
Diffstat (limited to 'test/libtest/driver/test_driver.1')
| -rw-r--r-- | test/libtest/driver/test_driver.1 | 308 |
1 files changed, 308 insertions, 0 deletions
diff --git a/test/libtest/driver/test_driver.1 b/test/libtest/driver/test_driver.1 new file mode 100644 index 0000000000000..a987002e8c670 --- /dev/null +++ b/test/libtest/driver/test_driver.1 @@ -0,0 +1,308 @@ +.\" Copyright (c) 2019 Joseph Koshy. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" This software is provided by Joseph Koshy ``as is'' and +.\" any express or implied warranties, including, but not limited to, the +.\" implied warranties of merchantability and fitness for a particular purpose +.\" are disclaimed. in no event shall Joseph Koshy be liable +.\" for any direct, indirect, incidental, special, exemplary, or consequential +.\" damages (including, but not limited to, procurement of substitute goods +.\" or services; loss of use, data, or profits; or business interruption) +.\" however caused and on any theory of liability, whether in contract, strict +.\" liability, or tort (including negligence or otherwise) arising in any way +.\" out of the use of this software, even if advised of the possibility of +.\" such damage. +.\" +.\" $Id$ +.\" +.Dd April 02, 2019 +.Dt TEST-DRIVER 1 +.Os +.Sh NAME +.Nm test-driver +.Nd scaffolding for executing +.Xr test 3 +based tests from the command-line +.Sh SYNOPSIS +.Nm test-executable +.Op Fl c Ar artefact-archive-name +.Op Fl l +.Op Fl n Ar run-name +.Op Fl p Ar search-path-directory +.Op Fl R Ar runtime-base-directory +.Op Fl s Ar execution-style +.Op Fl t Ar test-selector +.Op Fl T Ar seconds +.Op Fl v +.Sh DESCRIPTION +The +.Nm +library provides a +.Fn main +function that will execute the +.Xr test 3 +based tests in an executable according to the options specified +on the command-line. +The +.Nm +library usually used in conjunction with code generated by the +.Xr make-test-scaffolding 1 +utility. +.Pp +Test executables built using +.Nm +recognize the following command-line options: +.Bl -tag -width indent +.It Fl c Ar archive-name +If this option is specified, then the +.Nm +provided scaffolding will copy test outputs and other artefacts from +the test run to the archive named by argument +.Ar archive-name . +The format of the archive is specified by the path name suffix of the +artefact name. +The supported output formats are those supported by +.Xr libarchive 3 . +.It Fl l +If this option is specified, then the +.Nm +utility will list the selected tests and exit. +.It Fl n Ar run-name +Use the specified test run name in status messages and to name +any files and directories created during the test run. +If this option is not specified, then the base name of the test +executable is used. +.It Fl p Ar search-path-directory +Add the argument +.Ar search-path-directory +to the list of directories searched for by +.Xr test 3 +utility functions. +.It Fl R Ar runtime-base-directory +Set the runtime base directory to the directory specified by the +argument +.Ar runtime-base-directory . +Tests execute with their current directory set to a subdirectory +within this directory. +The path specified by argument +.Ar runtime-base-directory +must exist, and must name a directory. +.Pp +If this option is not specified, then the +.Ev TEST_TMPDIR +environment variable will be examined. +If set to a non-empty value, then its value will be used. +Otherwise, the value of the +.Ev TMPDIR +environment variable will be used, if non-empty. +If neither of the environment variables +.Ev TEST_TMPDIR +and +.Ev TMPDIR +contain a non-empty value, then the path +.Dq Pa /tmp +will be used. +.It Fl s Ar execution-style +Set the desired execution style to that specified by argument +.Ar execution-style . +Legal values for +.Ar execution-style +are: +.Bl -tag -width indent -compact +.It Li atf +Be compatible with +.Nx +.Xr atf 9 . +.It Li tap +Be compatible with TAP +.Pq Test Anything Protocol . +.It Li test +Be compatible with libtest (this test framework). +.El +The default is to use libtest semantics. +.It Fl t Ar test-selector +Select (or deselect) tests to execute according to the argument +.Ar test-selector . +.Pp +Test selectors are specified using the following syntax: +.Bl -tag -compact -width indent +.It Xo +.Op Li - Ns +.Li c : Ns Ar pattern +.Xc +Select test cases whose names match +.Ar pattern . +Selecting a test case will cause all of its contained +test functions to be selected. +.It Xo +.Op Li - Ns +.Li f : Ns Ar pattern +.Xc +Select test functions whose names match +.Ar pattern . +.It Xo +.Op Li - Ns +.Li t : Ns Ar pattern +.Xc +Select the test cases and test functions associated with +tags matching +.Ar pattern . +.It Xo +.Op Li - Ns +.Ar pattern +.Xc +If the +.Li c , +.Li f +or +.Li t +qualifiers were not specified, then the pattern is matched +against the names of test cases. +.El +The +.Ar pattern +fields of test selectors use shell wildcard syntax, as implemented by +.Xr fnmatch 3 . +.Pp +If no test selectors are specified then all the tests present in +the test executable will be run. +Otherwise, the test selectors specified are processed in the +order specified on the command line. +.Pp +A test selector that does not start with a +.Dq Li - +will add the entries that it matches to the currently selected list +of tests. +A test selector that starts with a +.Dq Li - +will remove the entries that it matches from the currently selected list +of tests. +.Pp +If at least one test selector was specified, and if the result of +applying the specified test selectors was an empty list +of tests, then the +.Nm +library will exit with an error message. +.It Fl T Ar seconds +Set the timeout for individual tests to +.Ar seconds . +If a test function fails to return with the specified number of seconds +then it is treated as having failed. +The default is to wait indefinitely for the test function to complete. +.It Fl v +Increase verbosity level by 1. +The default verbosity level is 0. +.El +.Ss Link-time Pre-requisites +The +.Nm +library expects the following symbols to be present in the +test executable it is linked with: +.Pp +.Bl -tag -width indent -compact +.It Xo +.Vt struct test_case_descriptor +.Va test_cases Ns [] +.Xc +An array of test cases descriptors. +Test case descriptors described by +.Xr test_case 5 . +.It Xo +.Vt int +.Va test_case_count +.Xc +The number of entries in the +.Va test_cases +array. +.El +.Ss Test Execution +At start up, the +.Fn main +function provided by +.Nm +will select tests (and test cases) to execute, based on the test +selection options specified. +.Pp +For each selected test case, test execution proceeds as follows: +.Bl -enum -compact +.It +The runtime directory for the test case is created. +.It +The test process forks, with test execution continuing in the +child. +.It +.Pq Child +The current directory of the process is changed to the runtime +directory. +.It +.Pq Child +The test case set up function is then executed. +If this function returns an error then test case execution is +aborted. +.It +.Pq Child +Each selected test function in the test case is then executed and +its status is output to stdout (or stderr) according to the test +execution style selected. +.It +.Pq Child +The test case tear down function is then executed. +.It +If test artefacts need to be preserved, then these are +copied to the specified archive. +.It +The test's runtime directory is then deleted. +.El +.Pp +After all test cases have been attempted, the +.Fn main +function exits with the exit code appropriate for the +test execution style selected. +.Sh EXAMPLES +To run all tests in the binary named +.Pa tc_example , +copying test artefacts to a +.Xr cpio 1 +archive named +.Pa /tmp/tc_example.cpio , +use: +.Bd -literal -offset indent +tc_example -c /tmp/tc_example.cpio +.Ed +.Pp +To execute tests in the test case +.Dq tc1 +alone, use: +.Bd -literal -offset indent +tc_example -t 'c:tc1' +.Ed +.Pp +To execute tests in the test case +.Dq tc1 +but not the test functions associated with tag +.Li tag1 , +use: +.Bd -literal -offset indent +tc_example -t 'c:tc1' -t '-t:tag1' +.Ed +.Sh DIAGNOSTICS +Test programs built with the +.Nm +library will exit with an exit code of 0 if all of the selected tests +passed when run, and with a non-zero exit code if an error +occurred during test execution. +.Sh SEE ALSO +.Xr make-test-scaffolding 1 , +.Xr fnmatch 3 , +.Xr libarchive 3 , +.Xr test 3 , +.Xr test_case 5 |