diff options
Diffstat (limited to 'doc/kyuafile.5.in')
| -rw-r--r-- | doc/kyuafile.5.in | 407 |
1 files changed, 407 insertions, 0 deletions
diff --git a/doc/kyuafile.5.in b/doc/kyuafile.5.in new file mode 100644 index 000000000000..06cb2dbc42a8 --- /dev/null +++ b/doc/kyuafile.5.in @@ -0,0 +1,407 @@ +.\" Copyright 2012 The Kyua Authors. +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions are +.\" met: +.\" +.\" * Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" * Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" * Neither the name of Google Inc. nor the names of its contributors +.\" may be used to endorse or promote products derived from this software +.\" without specific prior written permission. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.Dd July 3, 2015 +.Dt KYUAFILE 5 +.Os +.Sh NAME +.Nm Kyuafile +.Nd Test suite description files +.Sh SYNOPSIS +.Fn atf_test_program "string name" "[string metadata]" +.Fn current_kyuafile +.Fn fs.basename "string path" +.Fn fs.dirname "string path" +.Fn fs.exists "string path" +.Fn fs.files "string path" +.Fn fs.is_absolute "string path" +.Fn fs.join "string path" "string path" +.Fn include "string path" +.Fn plain_test_program "string name" "[string metadata]" +.Fn syntax "int version" +.Fn tap_test_program "string name" "[string metadata]" +.Fn test_suite "string name" +.Sh DESCRIPTION +A test suite is a collection of test programs and is represented by a +hierarchical layout of test binaries on the file system. +Any subtree of the file system can represent a test suite, provided that it +includes one or more +.Nm Ns s , +which are the test suite definition files. +.Pp +A +.Nm +is a Lua script whose purpose is to describe the structure of the test +suite it belongs to. +To do so, the script has access to a collection of special functions provided +by +.Xr kyua 1 +as described in +.Sx Helper functions . +.Ss File versioning +Every +.Nm +file starts with a call to +.Fn syntax "int version" . +This call determines the specific schema used by the file so that future +backwards-incompatible modifications to the file can be introduced. +.Pp +Any new +.Nm +file should set +.Fa version +to +.Sq 2 . +.Ss Test suite definition +If the +.Nm +registers any test programs, +the +.Nm +must define the name of the test suite the test programs belong to by using the +.Fn test_suite +function at the very beginning of the file. +.Pp +The test suite name provided in the +.Fn test_suite +call tells +.Xr kyua 1 +which set of configuration variables from +.Xr kyua.conf 5 +to pass to the test programs at run time. +.Ss Test program registration +A +.Nm +can register test programs by means of a variety of +.Fn *_test_program +functions, all of which take the name of a test program and a set of +optional metadata properties that describe such test program. +.Pp +The test programs to be registered must live in the current directory; in +other words, the various +.Fn *_test_program +calls cannot reference test programs in other directories. +The rationale for this is to force all +.Nm +files to be self-contained, and to simplify their internal representation. +.Pp +.Em ATF test programs +are those that use the +.Xr atf 7 +libraries. +They can be registered with the +.Fn atf_test_program +table constructor. +This function takes the +.Fa name +of the test program and a collection of optional metadata settings for all +the test cases in the test program. +Any metadata properties defined by the test cases themselves override the +metadata values defined here. +.Pp +.Em Plain test programs +are those that return 0 on success and non-0 on failure; in general, most test +programs (even those that use fancy unit-testing libraries) behave this way and +thus also qualify as plain test programs. +They can be registered with the +.Fn plain_test_program +table constructor. +This function takes the +.Fa name +of the test program, an optional +.Fa test_suite +name that overrides the global test suite name, and a collection of optional +metadata settings for the test program. +.Pp +.Em TAP test programs +are those that implement the Test Anything Protocol. +They can be registered with the +.Fn tap_test_program +table constructor. +This function takes the +.Fa name +of the test program and a collection of optional metadata settings for the +test program. +.Pp +The following metadata properties can be passed to any test program definition: +.Bl -tag -width XX -offset indent +.It Va allowed_architectures +Whitespace-separated list of machine architecture names allowed by the test. +If empty or not defined, the test is allowed to run on any machine +architecture. +.It Va allowed_platforms +Whitespace-separated list of machine platform names allowed by the test. +If empty or not defined, the test is allowed to run on any machine +platform. +.It Va custom.NAME +Custom variable defined by the test where +.Sq NAME +denotes the name of the variable. +These variables are useful to tag your tests with information specific to +your project. +The values of such variables are propagated all the way from the tests to the +results files and later to any generated reports. +.Pp +Note that if the name happens to have dashes or any other special characters +in it, you will have to use a special Lua syntax to define the property. +Refer to the +.Sx EXAMPLES +section below for clarification. +.It Va description +Textual description of the test. +.It Va is_exclusive +If true, indicates that this test program cannot be executed along any other +programs at the same time. +Test programs that affect global system state, such as those that modify the +value of a +.Xr sysctl 8 +setting, must set themselves as exclusive to prevent failures due to race +conditions. +Defaults to false. +.It Va required_configs +Whitespace-separated list of configuration variables that the test requires +to be defined before it can run. +.It Va required_disk_space +Amount of available disk space that the test needs to run successfully. +.It Va required_files +Whitespace-separated list of paths that the test requires to exist before +it can run. +.It Va required_memory +Amount of physical memory that the test needs to run successfully. +.It Va required_programs +Whitespace-separated list of basenames or absolute paths pointing to executable +binaries that the test requires to exist before it can run. +.It Va required_user +If empty, the test has no restrictions on the calling user for it to run. +If set to +.Sq unprivileged , +the test needs to not run as root. +If set to +.Sq root , +the test must run as root. +.It Va timeout +Amount of seconds that the test is allowed to execute before being killed. +.El +.Ss Recursion +To reference test programs in another subdirectory, a different +.Nm +must be created in that directory and it must be included into the original +.Nm +by means of the +.Fn include +function. +.Pp +.Fn include +may only be called with a relative path and with at most one directory +component. +This is by design: Kyua uses the file system structure as the layout of the +test suite definition. +Therefore, each subdirectory in a test suite must include its own +.Nm +and each +.Nm +can only descend into the +.Nm Ns s +of immediate subdirectories. +.Pp +If you need to source a +.Nm +located in disjoint parts of your file system namespace, you will have to +create a +.Sq shadow tree +using symbolic links and possibly helper +.Nm Ns s +to plug the various subdirectories together. +See the +.Sx EXAMPLES +section below for details. +.Pp +Note that each file is processed in its own Lua environment: there is no +mechanism to pass state from one file to the other. +The reason for this is that there is no such thing as a +.Dq top-level +.Nm +in a test suite: the user has to be able to run the test suite from any +directory in a given hierarchy, and this execution must not depend on files +that live in parent directories. +.Ss Top-level Kyuafile +Every system has a top directory into which test suites get installed. +The default is +.Pa __TESTSDIR__ . +Within this directory live test suites, each of which is in an independent +subdirectory. +Each subdirectory can be provided separately by independent third-party +packages. +.Pp +Kyua allows running all the installed test suites at once in order to +provide comprehensive cross-component reports. +In order to do this, there is a special file in the top directory that knows +how to inspect the subdirectories in search for other Kyuafiles and include +them. +.Pp +The +.Sx FILES +section includes more details on where this file lives. +.Ss Helper functions +The +.Sq base , +.Sq string , +and +.Sq table +Lua modules are fully available in the context of a +.Nm . +.Pp +The following extra functions are provided by Kyua: +.Bl -tag -width XX -offset indent +.It Ft string Fn current_kyuafile +Returns the absolute path to the current +.Nm . +.It Ft string Fn fs.basename "string path" +Returns the last component of the given path. +.It Ft string Fn fs.dirname "string path" +Returns the given path without its last component or a dot if the path has +a single component. +.It Ft bool Fn fs.exists "string path" +Checks if the given path exists. +If the path is not absolute, it is relative to the directory containing the +.Nm +in which the call to this function occurs. +.It Ft iterator Fn fs.files "string path" +Opens a directory for scanning of its entries. +The returned iterator yields an entry on each call, and the entry is simply +the filename. +If the path is not absolute, it is relative to the directory containing the +.Nm +in which the call to this function occurs. +.It Ft is_absolute Fn fs.is_absolute "string path" +Returns true if the given path is absolute; false otherwise. +.It Ft join Fn fs.join "string path" "string path" +Concatenates the two paths. +The second path cannot be absolute. +.El +.Sh FILES +.Bl -tag -width XX +.It Pa __TESTSDIR__/Kyuafile . +Top-level +.Nm +for the current system. +.It Pa __EGDIR__/Kyuafile.top . +Sample file to serve as a top-level +.Nm . +.El +.Sh EXAMPLES +The following +.Nm +is the simplest you can define. +It provides a test suite definition and registers a couple of different test +programs using different interfaces: +.Bd -literal -offset indent +syntax(2) + +test_suite('first') + +atf_test_program{name='integration_test'} +plain_test_program{name='legacy_test'} +.Ed +.Pp +The following example is a bit more elaborate. +It introduces some metadata properties to the test program definitions and +recurses into a couple of subdirectories: +.Bd -literal -offset indent +syntax(2) + +test_suite('second') + +plain_test_program{name='legacy_test', + allowed_architectures='amd64 i386', + required_files='/bin/ls', + timeout=30} + +tap_test_program{name='privileged_test', + required_user='root'} + +include('module-1/Kyuafile') +include('module-2/Kyuafile') +.Ed +.Pp +The syntax to define custom properties may be not obvious if their names +have any characters that make the property name not be a valid Lua identifier. +Dashes are just one example. +To set such properties, do something like this: +.Bd -literal -offset indent +syntax(2) + +test_suite('FreeBSD') + +plain_test_program{name='the_test', + ['custom.FreeBSD-Bug-Id']='category/12345'} +.Ed +.Ss Connecting disjoint test suites +Now suppose you had various test suites on your file system and you would +like to connect them together so that they could be executed and treated as +a single unit. +The test suites we would like to connect live under +.Pa /usr/tests , +.Pa /usr/local/tests +and +.Pa ~/local/tests . +.Pp +We cannot create a +.Nm +that references these because the +.Fn include +directive does not support absolute paths. +Instead, what we can do is create a shadow tree using symbolic links: +.Bd -literal -offset indent +$ mkdir ~/everything +$ ln -s /usr/tests ~/everything/system-tests +$ ln -s /usr/local/tests ~/everything/local-tests +$ ln -s ~/local/tests ~/everything/home-tests +.Ed +.Pp +And then we create an +.Pa ~/everything/Kyuafile +file to drive the execution of the integrated test suite: +.Bd -literal -offset indent +syntax(2) + +test_suite('test-all-the-things') + +include('system-tests/Kyuafile') +include('local-tests/Kyuafile') +include('home-tests/Kyuafile') +.Ed +.Pp +Or, simply, you could reuse the sample top-level +.Nm +to avoid having to manually craft the list of directories into which to +recurse: +.Bd -literal -offset indent +$ cp __EGDIR__/Kyuafile.top ~/everything/Kyuafile +.Ed +.Sh SEE ALSO +.Xr kyua 1 |