2 .\" Copyright (c) 2009-2019 Dell EMC Isilon http://www.isilon.com/
4 .\" Redistribution and use in source and binary forms, with or without
5 .\" modification, are permitted provided that the following conditions
7 .\" 1. Redistributions of source code must retain the above copyright
8 .\" notice(s), this list of conditions and the following disclaimer as
9 .\" the first lines of this file unmodified other than the possible
10 .\" addition of one or more copyright notices.
11 .\" 2. Redistributions in binary form must reproduce the above copyright
12 .\" notice(s), this list of conditions and the following disclaimer in the
13 .\" documentation and/or other materials provided with the distribution.
15 .\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY
16 .\" EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
17 .\" WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
18 .\" DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY
19 .\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
20 .\" (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
21 .\" SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
22 .\" CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
23 .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
24 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
34 .Nm KFAIL_POINT_CODE ,
35 .Nm KFAIL_POINT_CODE_FLAGS ,
36 .Nm KFAIL_POINT_CODE_COND ,
37 .Nm KFAIL_POINT_ERROR ,
38 .Nm KFAIL_POINT_EVAL ,
39 .Nm KFAIL_POINT_DECLARE ,
40 .Nm KFAIL_POINT_DEFINE ,
41 .Nm KFAIL_POINT_GOTO ,
42 .Nm KFAIL_POINT_RETURN ,
43 .Nm KFAIL_POINT_RETURN_VOID ,
44 .Nm KFAIL_POINT_SLEEP_CALLBACKS ,
49 .Fn KFAIL_POINT_CODE "parent" "name" "code"
50 .Fn KFAIL_POINT_CODE_FLAGS "parent" "name" "flags" "code"
51 .Fn KFAIL_POINT_CODE_COND "parent" "name" "cond" "flags" "code"
52 .Fn KFAIL_POINT_ERROR "parent" "name" "error_var"
53 .Fn KFAIL_POINT_EVAL "name" "code"
54 .Fn KFAIL_POINT_DECLARE "name"
55 .Fn KFAIL_POINT_DEFINE "parent" "name" "flags"
56 .Fn KFAIL_POINT_GOTO "parent" "name" "error_var" "label"
57 .Fn KFAIL_POINT_RETURN "parent" "name"
58 .Fn KFAIL_POINT_RETURN_VOID "parent" "name"
59 .Fn KFAIL_POINT_SLEEP_CALLBACKS "parent" "name" "pre_func" "pre_arg" "post_func" "post_arg" "code"
61 Fail points are used to add code points where errors may be injected
62 in a user controlled fashion.
63 Fail points provide a convenient wrapper around user-provided error
64 injection code, providing a
66 MIB, and a parser for that MIB that describes how the error
67 injection code should fire.
69 The base fail point macro is
73 is a sysctl tree (frequently
75 for kernel fail points, but various subsystems may wish to provide
76 their own fail point trees), and
78 is the name of the MIB in that tree, and
80 is the error injection code.
83 argument does not require braces, but it is considered good style to
84 use braces for any multi-line code arguments.
87 argument, the evaluation of
91 value set in the sysctl MIB.
94 .Fn KFAIL_POINT_CODE_FLAGS
97 argument which controls the fail point's behaviour.
98 This can be used to e.g., mark the fail point's context as non-sleepable,
101 action to be coerced to a busy wait.
102 The supported flags are:
103 .Bl -ohang -offset indent
104 .It FAIL_POINT_USE_TIMEOUT_PATH
105 Rather than sleeping on a
107 call, just fire the post-sleep function after a timeout fires.
108 .It FAIL_POINT_NONSLEEPABLE
109 Mark the fail point as being in a non-sleepable context, which coerces
117 .Fn KFAIL_POINT_CODE_COND
120 argument, which allows you to set the condition under which the fail point's
122 This is equivalent to:
125 KFAIL_POINT_CODE_FLAGS(...);
134 macros are wrappers around common error injection paths:
136 .It Fn KFAIL_POINT_RETURN parent name
138 .Sy KFAIL_POINT_CODE(..., return RETURN_VALUE)
139 .It Fn KFAIL_POINT_RETURN_VOID parent name
141 .Sy KFAIL_POINT_CODE(..., return)
142 .It Fn KFAIL_POINT_ERROR parent name error_var
144 .Sy KFAIL_POINT_CODE(..., error_var = RETURN_VALUE)
145 .It Fn KFAIL_POINT_GOTO parent name error_var label
147 .Sy KFAIL_POINT_CODE(..., { error_var = RETURN_VALUE; goto label;})
150 You can also introduce fail points by separating the declaration,
151 definition, and evaluation portions.
153 .It Fn KFAIL_POINT_DECLARE name
154 is used to declare the
157 .It Fn KFAIL_POINT_DEFINE parent name flags
158 defines and initializes the
162 .It Fn KFAIL_POINT_EVAL name code
163 is used at the point that the fail point is executed.
168 macros add sysctl MIBs where specified.
169 Many base kernel MIBs can be found in the
171 tree (referenced in code by
174 The sysctl variable may be set in a number of ways:
176 [<pct>%][<cnt>*]<type>[(args...)][-><more terms>]
179 The <type> argument specifies which action to take; it can be one of:
180 .Bl -tag -width ".Dv return"
182 Take no action (does not trigger fail point code)
184 Trigger fail point code with specified argument
186 Sleep the specified number of milliseconds
190 Break into the debugger, or trap if there is no debugger support
192 Print that the fail point executed
194 Threads sleep at the fail point until the fail point is set to
197 Thread yields the cpu when the fail point is evaluated
199 Similar to sleep, but busy waits the cpu.
200 (Useful in non-sleepable contexts.)
203 The <pct>% and <cnt>* modifiers prior to <type> control when
205 The <pct>% form (e.g. "1.2%") can be used to specify a
206 probability that <type> will execute.
207 This is a decimal in the range (0, 100] which can specify up to
209 The <cnt>* form (e.g. "5*") can be used to specify the number of
210 times <type> should be executed before this <term> is disabled.
211 Only the last probability and the last count are used if multiple
212 are specified, i.e. "1.2%2%" is the same as "2%".
213 When both a probability and a count are specified, the probability
214 is evaluated before the count, i.e. "2%5*" means "2% of the time,
215 but only 5 times total".
217 The operator -> can be used to express cascading terms.
218 If you specify <term1>-><term2>, it means that if <term1> does not
220 <term2> is evaluated.
221 For the purpose of this operator, the return() and print() operators
222 are the only types that cascade.
223 A return() term only cascades if the code executes, and a print()
224 term only cascades when passed a non-zero argument.
225 A pid can optionally be specified.
226 The fail point term is only executed when invoked by a process with a
230 .It Sy sysctl debug.fail_point.foobar="2.1%return(5)"
231 21/1000ths of the time, execute
233 with RETURN_VALUE set to 5.
234 .It Sy sysctl debug.fail_point.foobar="2%return(5)->5%return(22)"
235 2/100ths of the time, execute
237 with RETURN_VALUE set to 5.
238 If that does not happen, 5% of the time execute
240 with RETURN_VALUE set to 22.
241 .It Sy sysctl debug.fail_point.foobar="5*return(5)->0.1%return(22)"
242 For 5 times, return 5.
243 After that, 1/1000th of the time, return 22.
244 .It Sy sysctl debug.fail_point.foobar="0.1%5*return(5)"
245 Return 5 for 1 in 1000 executions, but only 5 times total.
246 .It Sy sysctl debug.fail_point.foobar="1%*sleep(50)"
247 1/100th of the time, sleep 50ms.
248 .It Sy sysctl debug.fail_point.foobar="1*return(5)[pid 1234]"
249 Return 5 once, when pid 1234 executes the fail point.
253 This manual page was written by
255 .An Matthew Bryan Aq Mt matthew.bryan@isilon.com
258 .An Zach Loafman Aq Mt zml@FreeBSD.org .
260 It is easy to shoot yourself in the foot by setting fail points too
261 aggressively or setting too many in combination.
264 to fail consistently is potentially harmful to uptime.
268 sysctl setting may not be appropriate in all situations.
271 does not verify whether the context is appropriate for calling
273 You can force it to evaluate a
277 action by specifying the
278 .Sy FAIL_POINT_NONSLEEPABLE
279 flag at the point the fail point is declared.