2 .\" Copyright (c) 2016-2018 Netflix, Inc.
3 .\" All rights reserved.
5 .\" Redistribution and use in source and binary forms, with or without
6 .\" modification, are permitted provided that the following conditions
8 .\" 1. Redistributions of source code must retain the above copyright
9 .\" notice, this list of conditions, and the following disclaimer,
10 .\" without modification, immediately at the beginning of the file.
11 .\" 2. The name of the author may not be used to endorse or promote products
12 .\" derived from this software without specific prior written permission.
14 .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
15 .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
16 .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
17 .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR
18 .\" ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
19 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
20 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
21 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
22 .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
23 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
33 .Nd statistics gathering
40 .Ss Stats Blob Template Management Functions
43 .Fa "const char *name"
47 .Fo stats_tpl_fetch_allocid
48 .Fa "const char *name"
54 .Fa "struct statsblob_tpl **tpl"
63 .Fo stats_tpl_sample_rates
64 .Fa "SYSCTL_HANDLER_ARGS"
67 .Fo stats_tpl_sample_rollthedice
68 .Fa "struct stats_tpl_sample_rate *rates"
70 .Fa "void *seed_bytes"
73 .Ft struct voistatspec
76 .Ft struct voistatspec
79 .Ft struct voistatspec
82 .Ft struct voistatspec
83 .Fo STATS_VSS_CRHIST<32|64>_LIN
89 .Ft struct voistatspec
90 .Fo STATS_VSS_CRHIST<32|64>_EXP
97 .Ft struct voistatspec
98 .Fo "STATS_VSS_CRHIST<32|64>_LINEXP"
105 .Ft struct voistatspec
106 .Fo "STATS_VSS_CRHIST<32|64>_USR"
107 .Fa Sy "HBKTS" Ns Pq Sy "CRBKT" Ns ( Em "lb" ) , "..." Pc ,
110 .Ft struct voistatspec
111 .Fo "STATS_VSS_DRHIST<32|64>_USR"
112 .Fa Sy "HBKTS" Ns Pq Sy "DRBKT" Ns ( Em "lb" , "ub" ) , "..." Pc ,
115 .Ft struct voistatspec
116 .Fo "STATS_VSS_DVHIST<32|64>_USR"
117 .Fa Sy "HBKTS" Ns Pq Sy "DVBKT" Ns ( Em "val" ) , "..." Pc ,
120 .Ft struct voistatspec
121 .Fo STATS_VSS_TDGSTCLUST<32|64>
126 .Fo stats_tpl_add_voistats
127 .Fa "uint32_t tpl_id"
129 .Fa "const char *voi_name"
130 .Fa "enum vsd_dtype voi_dtype"
132 .Fa "struct voistatspec *vss"
135 .Ss Stats Blob Data Gathering Functions
137 .Fo stats_voi_update_<abs|rel>_<dtype>
138 .Fa "struct statsblob *sb"
142 .Ss Stats Blob Utility Functions
143 .Ft struct statsblob *
145 .Fa "uint32_t tpl_id"
150 .Fa "struct statsblob *sb"
151 .Fa "uint32_t tpl_id"
156 .Fa "struct statsblob **dst"
157 .Fa "size_t dstmaxsz"
158 .Fa "struct statsblob *src"
162 .Fo stats_blob_destroy
163 .Fa "struct statsblob *sb"
166 .Fo stats_voistat_fetch_dptr
167 .Fa "struct statsblob *sb"
169 .Fa "enum voi_stype stype"
170 .Fa "enum vsd_dtype *retdtype"
171 .Fa "struct voistatdata **retvsd"
172 .Fa "size_t *retvsdsz"
175 .Fo stats_voistat_fetch_<dtype>
176 .Fa "struct statsblob *sb"
178 .Fa "enum voi_stype stype"
182 .Fo stats_blob_snapshot
183 .Fa "struct statsblob **dst"
184 .Fa "size_t dstmaxsz"
185 .Fa "struct statsblob *src"
190 .Fa "struct statsblob *sb"
191 .Fa "struct sbuf *buf"
192 .Fa "enum sb_str_fmt fmt"
196 .Fo stats_voistatdata_tostr
197 .Fa "const struct voistatdata *vsd"
198 .Fa "enum vsd_dtype dtype"
199 .Fa "enum sb_str_fmt fmt"
200 .Fa "struct sbuf *buf"
204 .Fn "\*(lp*stats_blob_visitcb_t\*(rp" "struct sb_visit *sbv" "void *usrctx"
207 .Fa "struct statsblob *sb"
208 .Fa "stats_blob_visitcb_t func"
214 framework facilitates real-time kernel and user space statistics gathering.
215 The framework is built around the
217 an object embedded within a contiguous memory allocation that is mostly opaque
218 to consumers and stores all required state.
221 object can itself be embedded within other objects either directly or indirectly
224 Objects or subsystems for which statistics are to be gathered are initialized
227 which acts as the blueprint for an arbitrary set of
228 Variables Of Interest (VOIs) and their associated statistics.
229 Each template defines a schema plus associated metadata, which are kept separate
230 to minimize the memory footprint of blobs.
232 Data gathering hook functions added at appropriate locations within the code
233 base of interest feed VOI data into the framework for processing.
239 header and opaque internal blob structure per the following diagram:
240 .Bd -literal -offset indent
241 ---------------------------------------------------------
243 | statsblob | opaque[] |
244 ---------------------------------------------------------
247 The publicly visible 8-byte header is defined as:
248 .Bd -literal -offset indent
260 specifies which API version the blob's
263 .Pq Dv STATS_ABI_V1 is the only version currently defined .
265 specifies the endianness of the blob's fields
272 for unknown endianness
275 specifies the size of the blob, while
277 specifies the size of the underlying memory allocation in which the
283 default to units of bytes, unless a flag is set in
285 that dictates otherwise.
287 Templates are constructed by associating arbitrary VOI IDs with a set of
288 statistics, where each statistic is specified using a
289 .Vt struct voistatspec
290 per the definition below:
291 .Bd -literal -offset indent
294 struct vss_hlpr_info *hlprinfo;
295 struct voistatdata *iv;
298 enum vsd_dtype vs_dtype : 8;
299 enum voi_stype stype : 8;
303 It is generally expected that consumers will not work with
304 .Vt struct voistatspec
305 directly, and instead use the
311 framework offers the following statistics for association with VOIs:
312 .Bl -tag -width ".Dv VS_STYPE_TDGST"
314 The sum of VOI values.
316 The maximum VOI value.
318 The minimum VOI value.
320 A static bucket histogram of VOI values, including a count of
321 .Dq out-of-band/bucket Dc
322 values which did not match any bucket.
323 Histograms can be specified as
324 .Dq Em C Ns ontinuous Em R Ns ange Dc
326 .Dq Em D Ns iscrete Em R Ns ange Dc
329 .Dq Em D Ns iscrete Em V Ns alue Dc
331 with 32 or 64 bit bucket counters, depending on the VOI semantics.
332 .It Dv VS_STYPE_TDGST
333 A dynamic bucket histogram of VOI values based on the t-digest method
334 .Po refer to the t-digest paper in the
341 .Dq visitor software design pattern Ns
342 -like scheme is employed to facilitate iterating over a blob's data without
343 concern for the blob's structure.
344 The data provided to visitor callback functions is encapsulated in
346 per the definition below:
347 .Bd -literal -offset indent
349 struct voistatdata *vs_data;
354 enum vsd_dtype voi_dtype : 8;
355 enum vsd_dtype vs_dtype : 8;
362 .Fn stats_tpl_sample_rates
364 .Fn stats_tpl_sample_rollthedice
366 .Vt struct stats_tpl_sample_rate
367 to encapsulate per-template sample rate information per the definition below:
368 .Bd -literal -offset indent
369 struct stats_tpl_sample_rate {
371 uint32_t tpl_sample_pct;
377 member holds the template's slot ID obtained from
380 .Fn stats_tpl_fetch_allocid .
383 member holds the template's sample rate as an integer percentage in the range
387 .Vt stats_tpl_sr_cb_t
388 conformant function pointer that is required as the
391 .Fn stats_tpl_sample_rates
393 .Bd -literal -offset indent
394 enum stats_tpl_sr_cb_action {
400 typedef int (*stats_tpl_sr_cb_t)(enum stats_tpl_sr_cb_action action,
401 struct stats_tpl_sample_rate **rates, int *nrates, void *ctx);
404 It is required that a conformant function:
407 Return an appropriate
409 on error, otherwise 0.
412 .Qq action == TPL_SR_*_GET ,
413 return the subsystem's rates list ptr and count, locked or unlocked as
417 .Qq action == TPL_SR_RUNLOCK ,
418 unlock the subsystem's rates list ptr and count.
420 .Qq action == TPL_SR_RLOCKED_GET
424 .Qq action == TPL_SR_PUT ,
425 update the subsystem's rates list ptr and count to the sysctl processed values
426 and return the inactive list details in
430 for garbage collection by
431 .Fn stats_tpl_sample_rates .
434 Where templates need to be referenced via textual means, for example via a MIB
435 variable, the following string based template spec formats can be used:
441 .Qq TCP_DEFAULT Qc Ns
453 The first form is the normative spec format generated by the framework, while
454 the second and third forms are convenience formats primarily for user input.
455 The use of inverted commas around the template name is optional.
459 framework exposes the following framework-specific variables in the
464 .Bl -tag -width "templates"
466 Read-only CSV list of registered templates in normative template spec form.
468 .Ss Template Management Functions
471 function allocates a new template with the specified unique name and returns its
472 runtime-stable template slot ID for use with other API functions.
475 argument is currently unused.
478 .Fn stats_tpl_fetch_allocid
479 function returns the runtime-stable template slot ID of any registered template
480 matching the specified name and hash.
484 function returns the pointer to the registered template object at the specified
488 .Fn stats_tpl_id2name
489 function returns the name of the registered template object at the specified
493 .Fn stats_tpl_sample_rates
494 function provides a generic handler for template sample rates management and
498 Subsystems can use this function to create a subsystem-specific
500 MIB variable that manages and reports subsystem-specific template sampling
502 Subsystems must supply a
503 .Vt stats_tpl_sr_cb_t
504 conformant function pointer as the sysctl's
506 which is a callback used to interact with the subsystem's stats template sample
508 Subsystems can optionally specify the sysctl's
510 as non-zero, which causes a zero-initialized allocation of arg2-sized contextual
511 memory to be heap-allocated and passed in to all subsystem callbacks made during
513 .Fn stats_tpl_sample_rates .
516 .Fn stats_tpl_sample_rollthedice
517 function makes a weighted random template selection from the supplied array of
518 template sampling rates.
519 The cumulative percentage of all sampling rates should not exceed 100.
520 If no seed is supplied, a PRNG is used to generate a true random number so that
521 every selection is independent.
522 If a seed is supplied, selection will be made randomly across different seeds, but
523 deterministically given the same seed.
526 .Fn stats_tpl_add_voistats
527 function is used to add a VOI and associated set of statistics to the registered
528 template object at the specified template slot ID.
529 The set of statistics is passed as an array of
530 .Vt struct voistatspec
531 which can be initialized using the
533 helper macros or manually for non-standard use cases.
538 count of array elements can be determined by passing
545 flag can be passed to configure the VOI for use with
546 .Fn stats_voi_update_rel_<dtype> ,
547 which entails maintaining an extra 8 bytes of state in the blob at each update.
548 .Ss Data Gathering Functions
550 .Fn stats_voi_update_abs_<dtype>
552 .Fn stats_voi_update_rel_<dtype>
553 functions both update all the statistics associated with the VOI identified by
559 as an absolute value, whereas the
563 as a value relative to that of the previous update function call, by adding it
564 to the previous value and using the result for the update.
565 Relative updates are only possible for VOIs that were added to the template with
569 .Fn stats_tpl_add_voistats .
570 .Ss Utility Functions
573 function allocates and initializes a new blob based on the registered template
574 object at the specified template slot ID.
578 function initializes a new blob in an existing memory allocation based on the
579 registered template object at the specified template slot ID.
583 function duplicates the
593 .Dv SB_CLONE_ALLOCDST
594 flag can be passed to instruct the function to allocate a new blob of
595 appropriate size into which to clone
597 storing the new pointer in
600 .Dv SB_CLONE_USRDSTNOFAULT
603 flags can be set to respectively signal that
604 .Xr copyout_nofault 9
607 should be used because
609 is a user space address.
612 .Fn stats_blob_snapshot
617 and then performs any additional functions required to produce a coherent
619 The flags interpreted by
622 .Fn stats_blob_snapshot .
625 flag can be used to effect a reset of the
627 blob's statistics after a snapshot is successfully taken.
630 .Fn stats_blob_destroy
631 function destroys a blob previously created with
632 .Fn stats_blob_alloc ,
635 .Fn stats_blob_snapshot .
639 function allows the caller to iterate over the contents of a blob.
640 The callback function
642 is called for every VOI and statistic in the blob, passing a
644 and the user context argument
646 to the callback function.
649 passed to the callback function may have one or more of the following flags set
652 struct member to provide useful metadata about the iteration:
655 .Dv SB_IT_FIRST_VOI ,
657 .Dv SB_IT_FIRST_VOISTAT ,
658 .Dv SB_IT_LAST_VOISTAT ,
661 .Dv SB_IT_NULLVOISTAT .
662 Returning a non-zero value from the callback function terminates the iteration.
666 renders a string representation of a blob into the
669 Currently supported render formats are
670 .Dv SB_STRFMT_FREEFORM
675 flag can be passed to render version specific opaque implementation detail for
676 debugging or string-to-binary blob reconstruction purposes.
679 flag can be passed to render template metadata into the string representation,
680 using the blob's template hash to lookup the corresponding template.
683 .Fn stats_voistatdata_tostr
684 renders a string representation of an individual statistic's data into the
687 The same render formats supported by the
689 function can be specified, and the
691 boolean has the same meaning as the
696 .Fn stats_voistat_fetch_dptr
697 function returns an internal blob pointer to the specified
699 statistic data for the VOI
702 .Fn stats_voistat_fetch_<dtype>
703 functions are convenience wrappers around
704 .Fn stats_voistat_fetch_dptr
705 to perform the extraction for simple data types.
706 .Sh IMPLEMENTATION NOTES
707 The following notes apply to STATS_ABI_V1 format statsblobs.
708 .Ss Space-Time Complexity
709 Blobs are laid out as three distinct memory regions following the header:
710 .Bd -literal -offset indent
711 ------------------------------------------------------
712 | struct | struct | struct | struct |
713 | statsblobv1 | voi [] | voistat [] | voistatdata [] |
714 ------------------------------------------------------
717 Blobs store VOI and statistic blob state
725 in sparse arrays, using the
730 This allows O(1) access to any voi/voistat pair in the blob, at the expense of
731 8 bytes of wasted memory per vacant slot for templates which do not specify
732 contiguously numbered VOIs and/or statistic types.
733 Data storage for statistics is only allocated for non-vacant slot pairs.
735 To provide a concrete example, a blob with the following specification:
738 Two VOIs; ID 0 and 2; added to the template in that order
740 VOI 0 is of data type
744 to enable support for relative updates using
745 .Fn stats_voi_update_rel_<dtype> ,
748 statistic associated with it.
750 VOI 2 is of data type
756 statistics associated with it.
759 would have the following memory layout:
761 --------------------------------------
762 | header | struct statsblobv1, 32 bytes
763 |------------------------------------|
764 | voi[0] | struct voi, 8 bytes
765 | voi[1] (vacant) | struct voi, 8 bytes
766 | voi[2] | struct voi, 8 bytes
767 |------------------------------------|
768 | voi[2]voistat[VOISTATE] (vacant) | struct voistat, 8 bytes
769 | voi[2]voistat[SUM] | struct voistat, 8 bytes
770 | voi[2]voistat[MAX] | struct voistat, 8 bytes
771 | voi[0]voistat[VOISTATE] | struct voistat, 8 bytes
772 | voi[0]voistat[SUM] (vacant) | struct voistat, 8 bytes
773 | voi[0]voistat[MAX] (vacant) | struct voistat, 8 bytes
774 | voi[0]voistat[MIN] | struct voistat, 8 bytes
775 |------------------------------------|
776 | voi[2]voistat[SUM]voistatdata | struct voistatdata_int32, 4 bytes
777 | voi[2]voistat[MAX]voistatdata | struct voistatdata_int32, 4 bytes
778 | voi[0]voistat[VOISTATE]voistatdata | struct voistatdata_numeric, 8 bytes
779 | voi[0]voistat[MIN]voistatdata | struct voistatdata_int64, 8 bytes
780 --------------------------------------
784 When rendered to string format using
785 .Fn stats_blob_tostr ,
787 .Dv SB_STRFMT_FREEFORM
791 flag, the rendered output is:
793 struct statsblobv1@0x8016250a0, abi=1, endian=1, maxsz=136, cursz=136, \\
794 created=6294158585626144, lastrst=6294158585626144, flags=0x0000, \\
795 stats_off=56, statsdata_off=112, tplhash=2994056564
796 vois[0]: id=0, name="", flags=0x0001, dtype=INT_S64, voistatmaxid=3, \\
798 vois[0]stat[0]: stype=VOISTATE, flags=0x0000, dtype=VOISTATE, \\
801 vois[0]stat[1]: stype=-1
802 vois[0]stat[2]: stype=-1
803 vois[0]stat[3]: stype=MIN, flags=0x0000, dtype=INT_S64, \\
805 voistatdata: 9223372036854775807
807 vois[2]: id=2, name="", flags=0x0000, dtype=INT_U32, voistatmaxid=2, \\
809 vois[2]stat[0]: stype=-1
810 vois[2]stat[1]: stype=SUM, flags=0x0000, dtype=INT_U32, dsz=4, \\
813 vois[2]stat[2]: stype=MAX, flags=0x0000, dtype=INT_U32, dsz=4, \\
820 present in the rendered output above indicates a manual line break inserted to
821 keep the man page within 80 columns and is not part of the actual output.
825 framework does not provide any concurrency protection at the individual blob
826 level, instead requiring that consumers guarantee mutual exclusion when calling
827 API functions that reference a non-template blob.
829 The list of templates is protected with a
833 rw lock in user space to support concurrency between template management and
834 blob initialization operations.
837 returns a runtime-stable template slot ID on success, or a negative errno on
839 -EINVAL is returned if any problems are detected with the arguments.
840 -EEXIST is returned if an existing template is registered with the same name.
841 -ENOMEM is returned if a required memory allocation fails.
843 .Fn stats_tpl_fetch_allocid
844 returns a runtime-stable template slot ID, or negative errno on failure.
845 -ESRCH is returned if no registered template matches the specified name and/or
849 returns 0 on success, or ENOENT if an invalid
853 .Fn stats_tpl_id2name
854 returns 0 on success, or an errno on failure.
855 EOVERFLOW is returned if the length of
859 is too short to hold the template's name.
860 ENOENT is returned if an invalid
864 .Fn stats_tpl_sample_rollthedice
865 returns a valid template slot id selected from
867 or -1 if a NULL selection was made, that is no stats collection this roll.
869 .Fn stats_tpl_add_voistats
870 return 0 on success, or an errno on failure.
871 EINVAL is returned if any problems are detected with the arguments.
872 EFBIG is returned if the resulting blob would have exceeded the maximum size.
873 EOPNOTSUPP is returned if an attempt is made to add more VOI stats to a
874 previously configured VOI.
875 ENOMEM is returned if a required memory allocation fails.
877 .Fn stats_voi_update_abs_<dtype>
879 .Fn stats_voi_update_rel_<dtype>
880 return 0 on success, or EINVAL if any problems are detected with the arguments.
883 returns 0 on success, or an errno on failure.
884 EINVAL is returned if any problems are detected with the arguments.
885 EOVERFLOW is returned if the template blob's
889 of the blob being initialized.
892 returns a pointer to a newly allocated and initialized blob based on the
893 specified template with slot ID
895 or NULL if the memory allocation failed.
899 .Fn stats_blob_snapshot
900 return 0 on success, or an errno on failure.
901 EINVAL is returned if any problems are detected with the arguments.
902 ENOMEM is returned if the SB_CLONE_ALLOCDST flag was specified and the memory
906 EOVERFLOW is returned if the src blob's
915 returns 0 on success, or EINVAL if any problems are detected with the arguments.
919 .Fn stats_voistatdata_tostr
920 return 0 on success, or an errno on failure.
921 EINVAL is returned if any problems are detected with the arguments, otherwise
922 any error returned by
928 .Fn stats_voistat_fetch_dptr
929 returns 0 on success, or EINVAL if any problems are detected with the arguments.
931 .Fn stats_voistat_fetch_<dtype>
932 returns 0 on success, or an errno on failure.
933 EINVAL is returned if any problems are detected with the arguments.
934 EFTYPE is returned if the requested data type does not match the blob's data
935 type for the specified
948 .%T "Computing Extremely Accurate Quantiles Using t-digests"
949 .%U "https://github.com/tdunning/t-digest/raw/master/docs/t-digest-paper/histo.pdf"
954 framework first appeared in
960 framework and this manual page were written by
961 .An Lawrence Stewart Aq lstewart@FreeBSD.org
962 and sponsored by Netflix, Inc.
964 Granularity of timing-dependent network statistics, in particular TCP_RTT,
968 To minimize the measurement error avoid using HZ lower than 1000.