1 .\" Copyright (c) 2008 The NetBSD Foundation, Inc.
2 .\" All rights reserved.
4 .\" Redistribution and use in source and binary forms, with or without
5 .\" modification, are permitted provided that the following conditions
7 .\" 1. Redistributions of source code must retain the above copyright
8 .\" notice, this list of conditions and the following disclaimer.
9 .\" 2. Redistributions in binary form must reproduce the above copyright
10 .\" notice, this list of conditions and the following disclaimer in the
11 .\" documentation and/or other materials provided with the distribution.
13 .\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND
14 .\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
15 .\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
16 .\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
17 .\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY
18 .\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
19 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
20 .\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
21 .\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
22 .\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
23 .\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
24 .\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
29 .Nm atf_add_test_case ,
34 .Nm atf_expect_death ,
38 .Nm atf_expect_signal ,
39 .Nm atf_expect_timeout ,
44 .Nm atf_require_prog ,
48 .Nd POSIX shell API to write ATF-based test programs
55 .Qq expected_expression
77 .Nm atf_expect_timeout
98 provides a simple but powerful interface to easily write test programs in
99 the POSIX shell language.
100 These are extremely helpful given that they are trivial to write due to the
101 language simplicity and the great deal of available external tools, so they
102 are often ideal to test other applications at the user level.
104 Test programs written using this library must be run using the
106 interpreter by putting the following on their very first line:
107 .Bd -literal -offset indent
108 #! /usr/bin/env atf-sh
111 Shell-based test programs always follow this template:
112 .Bd -literal -offset indent
115 ... first test case's header ...
118 ... first test case's body ...
121 atf_test_case tc2 cleanup
123 ... second test case's header ...
126 ... second test case's body ...
129 ... second test case's cleanup ...
132 .Ns ... additional test cases ...
134 atf_init_test_cases() {
135 atf_add_test_case tc1
136 atf_add_test_case tc2
137 ... add additional test cases ...
140 .Ss Definition of test cases
141 Test cases have an identifier and are composed of three different parts:
142 the header, the body and an optional cleanup routine, all of which are
144 .Xr atf-test-case 4 .
145 To define test cases, one can use the
147 function, which takes a first parameter specifiying the test case's
148 name and instructs the library to set things up to accept it as a valid
150 The second parameter is optional and, if provided, must be
152 providing this parameter allows defining a cleanup routine for the test
154 It is important to note that this function
156 set the test case up for execution when the program is run.
157 In order to do so, a later registration is needed through the
158 .Nm atf_add_test_case
160 .Sx Program initialization .
162 Later on, one must define the three parts of the body by providing two
163 or three functions (remember that the cleanup routine is optional).
164 These functions are named after the test case's identifier, and are
165 .Nm \*(Ltid\*(Gt_head ,
166 .Nm \*(Ltid\*(Gt_body
168 .Nm \*(Ltid\*(Gt_cleanup .
169 None of these take parameters when executed.
170 .Ss Program initialization
171 The test program must define an
172 .Nm atf_init_test_cases
173 function, which is in charge of registering the test cases that will be
174 executed at run time by using the
175 .Nm atf_add_test_case
176 function, which takes the name of a test case as its single parameter.
177 This main function should not do anything else, except maybe sourcing
178 auxiliary source files that define extra variables and functions.
179 .Ss Configuration variables
180 The test case has read-only access to the current configuration variables
186 The former takes a single parameter specifying a variable name and returns
187 a boolean indicating whether the variable is defined or not.
188 The latter can take one or two parameters.
189 If it takes only one, it specifies the variable from which to get the
190 value, and this variable must be defined.
191 If it takes two, the second one specifies a default value to be returned
192 if the variable is not available.
193 .Ss Access to the source directory
194 It is possible to get the path to the test case's source directory from
195 anywhere in the test program by using the
198 It is interesting to note that this can be used inside
199 .Nm atf_init_test_cases
200 to silently include additional helper files from the source directory.
201 .Ss Requiring programs
204 meta-data variable available in the header only, one can also check for
205 additional programs in the test case's body by using the
207 function, which takes the base name or full path of a single binary.
208 Relative paths are forbidden.
209 If it is not found, the test case will be automatically skipped.
210 .Ss Test case finalization
211 The test case finalizes either when the body reaches its end, at which
212 point the test is assumed to have
214 or at any explicit call to
219 These three functions terminate the execution of the test case immediately.
220 The cleanup routine will be processed afterwards in a completely automated
221 way, regardless of the test case's termination reason.
224 does not take any parameters.
228 take a single string parameter that describes why the test case failed or
229 was skipped, respectively.
230 It is very important to provide a clear error message in both cases so that
231 the user can quickly know why the test did not pass.
233 Everything explained in the previous section changes when the test case
234 expectations are redefined by the programmer.
236 Each test case has an internal state called
238 that describes what the test case expectations are at any point in time.
239 The value of this property can change during execution by any of:
240 .Bl -tag -width indent
241 .It Nm atf_expect_death Qo reason Qc Qo ... Qc
242 Expects the test case to exit prematurely regardless of the nature of the
244 .It Nm atf_expect_exit Qo exitcode Qc Qo reason Qc Qo ... Qc
245 Expects the test case to exit cleanly.
250 the runtime engine will validate that the exit code of the test case
251 matches the one provided in this call.
252 Otherwise, the exact value will be ignored.
253 .It Nm atf_expect_fail Qo reason Qc
254 Any failure raised in this mode is recorded, but such failures do not report
255 the test case as failed; instead, the test case finalizes cleanly and is
257 .Sq expected failure ;
258 this report includes the provided
261 If no error is raised while running in this mode, then the test case is
265 This mode is useful to reproduce actual known bugs in tests.
266 Whenever the developer fixes the bug later on, the test case will start
267 reporting a failure, signaling the developer that the test case must be
268 adjusted to the new conditions.
269 In this situation, it is useful, for example, to set
271 as the bug number for tracking purposes.
272 .It Nm atf_expect_pass
273 This is the normal mode of execution.
274 In this mode, any failure is reported as such to the user and the test case
277 .It Nm atf_expect_signal Qo signo Qc Qo reason Qc Qo ... Qc
278 Expects the test case to terminate due to the reception of a signal.
283 the runtime engine will validate that the signal that terminated the test
284 case matches the one provided in this call.
285 Otherwise, the exact value will be ignored.
286 .It Nm atf_expect_timeout Qo reason Qc Qo ... Qc
287 Expects the test case to execute for longer than its timeout.
289 .Ss Helper functions for common checks
290 .Bl -tag -width indent
291 .It Nm atf_check Qo [options] Qc Qo command Qc Qo [args] Qc
292 Executes a command, performs checks on its exit code and its output, and
293 fails the test case if any of the checks is not successful.
294 This function is particularly useful in integration tests that verify the
295 correct functioning of a binary.
297 Internally, this function is just a wrapper over the
299 tool (whose manual page provides all details on the calling syntax).
300 You should always use the
302 function instead of the
304 tool in your scripts; the latter is not even in the path.
305 .It Nm atf_check_equal Qo expected_expression Qc Qo actual_expression Qc
306 This function takes two expressions, evaluates them and, if their
307 results differ, aborts the test case with an appropriate failure message.
308 The common style is to put the expected value in the first parameter and the
309 actual value in the second parameter.
312 The following shows a complete test program with a single test case that
313 validates the addition operator:
314 .Bd -literal -offset indent
315 atf_test_case addition
317 atf_set "descr" "Sample tests for the addition operator"
320 atf_check_equal 0 $((0 + 0))
321 atf_check_equal 1 $((0 + 1))
322 atf_check_equal 1 $((1 + 0))
324 atf_check_equal 2 $((1 + 1))
326 atf_check_equal 300 $((100 + 200))
329 atf_init_test_cases() {
330 atf_add_test_case addition
334 This other example shows how to include a file with extra helper functions
336 .Bd -literal -offset indent
337 .Ns ... definition of test cases ...
339 atf_init_test_cases() {
340 . $(atf_get_srcdir)/helper_functions.sh
342 atf_add_test_case foo1
343 atf_add_test_case foo2
347 This example demonstrates the use of the very useful
350 .Bd -literal -offset indent
351 # Check for silent output
352 atf_check -s exit:0 -o empty -e empty 'true'
354 # Check for silent output and failure
355 atf_check -s exit:1 -o empty -e empty 'false'
357 # Check for known stdout and silent stderr
359 atf_check -s exit:0 -o file:expout -e empty 'echo foo'
361 # Generate a file for later inspection
362 atf_check -s exit:0 -o save:stdout -e empty 'ls'
363 grep foo ls || atf_fail "foo file not found in listing"
365 # Or just do the match along the way
366 atf_check -s exit:0 -o match:"^foo$" -e empty 'ls'
371 .Xr atf-test-program 1 ,