1 .\" Copyright (c) 2008 The NetBSD Foundation, Inc.
2 .\" All rights reserved.
4 .\" Redistribution and use in source and binary forms, with or without
5 .\" modification, are permitted provided that the following conditions
7 .\" 1. Redistributions of source code must retain the above copyright
8 .\" notice, this list of conditions and the following disclaimer.
9 .\" 2. Redistributions in binary form must reproduce the above copyright
10 .\" notice, this list of conditions and the following disclaimer in the
11 .\" documentation and/or other materials provided with the distribution.
13 .\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND
14 .\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
15 .\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
16 .\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
17 .\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY
18 .\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
19 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
20 .\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
21 .\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
22 .\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
23 .\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
24 .\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
29 .Nm atf_add_test_case ,
34 .Nm atf_expect_death ,
38 .Nm atf_expect_signal ,
39 .Nm atf_expect_timeout ,
43 .Nm atf_init_test_cases ,
45 .Nm atf_require_prog ,
49 .Nd POSIX shell API to write ATF-based test programs
56 .Qq expected_expression
78 .Nm atf_expect_timeout
86 .Nm atf_init_test_cases
101 provides a simple but powerful interface to easily write test programs in
102 the POSIX shell language.
103 These are extremely helpful given that they are trivial to write due to the
104 language simplicity and the great deal of available external tools, so they
105 are often ideal to test other applications at the user level.
107 Test programs written using this library must be run using the
109 interpreter by putting the following on their very first line:
110 .Bd -literal -offset indent
111 #! /usr/bin/env atf-sh
114 Shell-based test programs always follow this template:
115 .Bd -literal -offset indent
118 ... first test case's header ...
121 ... first test case's body ...
124 atf_test_case tc2 cleanup
126 ... second test case's header ...
129 ... second test case's body ...
132 ... second test case's cleanup ...
135 \&... additional test cases ...
137 atf_init_test_cases() {
138 atf_add_test_case tc1
139 atf_add_test_case tc2
140 ... add additional test cases ...
143 .Ss Definition of test cases
144 Test cases have an identifier and are composed of three different parts:
145 the header, the body and an optional cleanup routine, all of which are
147 .Xr atf-test-case 4 .
148 To define test cases, one can use the
150 function, which takes a first parameter specifying the test case's
151 name and instructs the library to set things up to accept it as a valid
153 The second parameter is optional and, if provided, must be
155 providing this parameter allows defining a cleanup routine for the test
157 It is important to note that this function
159 set the test case up for execution when the program is run.
160 In order to do so, a later registration is needed through the
161 .Nm atf_add_test_case
163 .Sx Program initialization .
165 Later on, one must define the three parts of the body by providing two
166 or three functions (remember that the cleanup routine is optional).
167 These functions are named after the test case's identifier, and are
168 .Nm \*(Ltid\*(Gt_head ,
169 .Nm \*(Ltid\*(Gt_body
171 .Nm \*(Ltid\*(Gt_cleanup .
172 None of these take parameters when executed.
173 .Ss Program initialization
174 The test program must define an
175 .Nm atf_init_test_cases
176 function, which is in charge of registering the test cases that will be
177 executed at run time by using the
178 .Nm atf_add_test_case
179 function, which takes the name of a test case as its single parameter.
180 This main function should not do anything else, except maybe sourcing
181 auxiliary source files that define extra variables and functions.
182 .Ss Configuration variables
183 The test case has read-only access to the current configuration variables
189 The former takes a single parameter specifying a variable name and returns
190 a boolean indicating whether the variable is defined or not.
191 The latter can take one or two parameters.
192 If it takes only one, it specifies the variable from which to get the
193 value, and this variable must be defined.
194 If it takes two, the second one specifies a default value to be returned
195 if the variable is not available.
196 .Ss Access to the source directory
197 It is possible to get the path to the test case's source directory from
198 anywhere in the test program by using the
201 It is interesting to note that this can be used inside
202 .Nm atf_init_test_cases
203 to silently include additional helper files from the source directory.
204 .Ss Requiring programs
207 meta-data variable available in the header only, one can also check for
208 additional programs in the test case's body by using the
210 function, which takes the base name or full path of a single binary.
211 Relative paths are forbidden.
212 If it is not found, the test case will be automatically skipped.
213 .Ss Test case finalization
214 The test case finalizes either when the body reaches its end, at which
215 point the test is assumed to have
217 or at any explicit call to
222 These three functions terminate the execution of the test case immediately.
223 The cleanup routine will be processed afterwards in a completely automated
224 way, regardless of the test case's termination reason.
227 does not take any parameters.
231 take a single string parameter that describes why the test case failed or
232 was skipped, respectively.
233 It is very important to provide a clear error message in both cases so that
234 the user can quickly know why the test did not pass.
236 Everything explained in the previous section changes when the test case
237 expectations are redefined by the programmer.
239 Each test case has an internal state called
241 that describes what the test case expectations are at any point in time.
242 The value of this property can change during execution by any of:
243 .Bl -tag -width indent
244 .It Nm atf_expect_death Qo reason Qc Qo ... Qc
245 Expects the test case to exit prematurely regardless of the nature of the
247 .It Nm atf_expect_exit Qo exitcode Qc Qo reason Qc Qo ... Qc
248 Expects the test case to exit cleanly.
253 the runtime engine will validate that the exit code of the test case
254 matches the one provided in this call.
255 Otherwise, the exact value will be ignored.
256 .It Nm atf_expect_fail Qo reason Qc
257 Any failure raised in this mode is recorded, but such failures do not report
258 the test case as failed; instead, the test case finalizes cleanly and is
260 .Sq expected failure ;
261 this report includes the provided
264 If no error is raised while running in this mode, then the test case is
268 This mode is useful to reproduce actual known bugs in tests.
269 Whenever the developer fixes the bug later on, the test case will start
270 reporting a failure, signaling the developer that the test case must be
271 adjusted to the new conditions.
272 In this situation, it is useful, for example, to set
274 as the bug number for tracking purposes.
275 .It Nm atf_expect_pass
276 This is the normal mode of execution.
277 In this mode, any failure is reported as such to the user and the test case
280 .It Nm atf_expect_signal Qo signo Qc Qo reason Qc Qo ... Qc
281 Expects the test case to terminate due to the reception of a signal.
286 the runtime engine will validate that the signal that terminated the test
287 case matches the one provided in this call.
288 Otherwise, the exact value will be ignored.
289 .It Nm atf_expect_timeout Qo reason Qc Qo ... Qc
290 Expects the test case to execute for longer than its timeout.
292 .Ss Helper functions for common checks
293 .Bl -tag -width indent
294 .It Nm atf_check Qo [options] Qc Qo command Qc Qo [args] Qc
295 Executes a command, performs checks on its exit code and its output, and
296 fails the test case if any of the checks is not successful.
297 This function is particularly useful in integration tests that verify the
298 correct functioning of a binary.
300 Internally, this function is just a wrapper over the
302 tool (whose manual page provides all details on the calling syntax).
303 You should always use the
305 function instead of the
307 tool in your scripts; the latter is not even in the path.
308 .It Nm atf_check_equal Qo expected_expression Qc Qo actual_expression Qc
309 This function takes two expressions, evaluates them and, if their
310 results differ, aborts the test case with an appropriate failure message.
311 The common style is to put the expected value in the first parameter and the
312 actual value in the second parameter.
315 The following shows a complete test program with a single test case that
316 validates the addition operator:
317 .Bd -literal -offset indent
318 atf_test_case addition
320 atf_set "descr" "Sample tests for the addition operator"
323 atf_check_equal 0 $((0 + 0))
324 atf_check_equal 1 $((0 + 1))
325 atf_check_equal 1 $((1 + 0))
327 atf_check_equal 2 $((1 + 1))
329 atf_check_equal 300 $((100 + 200))
332 atf_init_test_cases() {
333 atf_add_test_case addition
337 This other example shows how to include a file with extra helper functions
339 .Bd -literal -offset indent
340 \&... definition of test cases ...
342 atf_init_test_cases() {
343 . $(atf_get_srcdir)/helper_functions.sh
345 atf_add_test_case foo1
346 atf_add_test_case foo2
350 This example demonstrates the use of the very useful
353 .Bd -literal -offset indent
354 # Check for silent output
355 atf_check -s exit:0 -o empty -e empty 'true'
357 # Check for silent output and failure
358 atf_check -s exit:1 -o empty -e empty 'false'
360 # Check for known stdout and silent stderr
362 atf_check -s exit:0 -o file:expout -e empty 'echo foo'
364 # Generate a file for later inspection
365 atf_check -s exit:0 -o save:stdout -e empty 'ls'
366 grep foo ls || atf_fail "foo file not found in listing"
368 # Or just do the match along the way
369 atf_check -s exit:0 -o match:"^foo$" -e empty 'ls'
374 .Xr atf-test-program 1 ,