2 .\" Copyright (c) 2016-2018 Netflix, Inc.
3 .\" All rights reserved.
5 .\" Redistribution and use in source and binary forms, with or without
6 .\" modification, are permitted provided that the following conditions
8 .\" 1. Redistributions of source code must retain the above copyright
9 .\" notice, this list of conditions, and the following disclaimer,
10 .\" without modification, immediately at the beginning of the file.
11 .\" 2. The name of the author may not be used to endorse or promote products
12 .\" derived from this software without specific prior written permission.
14 .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
15 .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
16 .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
17 .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR
18 .\" ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
19 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
20 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
21 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
22 .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
23 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
31 .Nd statistics gathering
38 .Ss Stats Blob Template Management Functions
41 .Fa "const char *name"
45 .Fo stats_tpl_fetch_allocid
46 .Fa "const char *name"
52 .Fa "struct statsblob_tpl **tpl"
61 .Fo stats_tpl_sample_rates
62 .Fa "SYSCTL_HANDLER_ARGS"
65 .Fo stats_tpl_sample_rollthedice
66 .Fa "struct stats_tpl_sample_rate *rates"
68 .Fa "void *seed_bytes"
71 .Ft struct voistatspec
74 .Ft struct voistatspec
77 .Ft struct voistatspec
80 .Ft struct voistatspec
81 .Fo STATS_VSS_CRHIST<32|64>_LIN
87 .Ft struct voistatspec
88 .Fo STATS_VSS_CRHIST<32|64>_EXP
95 .Ft struct voistatspec
96 .Fo "STATS_VSS_CRHIST<32|64>_LINEXP"
103 .Ft struct voistatspec
104 .Fo "STATS_VSS_CRHIST<32|64>_USR"
105 .Fa Sy "HBKTS" Ns Pq Sy "CRBKT" Ns ( Em "lb" ) , "..." Pc ,
108 .Ft struct voistatspec
109 .Fo "STATS_VSS_DRHIST<32|64>_USR"
110 .Fa Sy "HBKTS" Ns Pq Sy "DRBKT" Ns ( Em "lb" , "ub" ) , "..." Pc ,
113 .Ft struct voistatspec
114 .Fo "STATS_VSS_DVHIST<32|64>_USR"
115 .Fa Sy "HBKTS" Ns Pq Sy "DVBKT" Ns ( Em "val" ) , "..." Pc ,
118 .Ft struct voistatspec
119 .Fo STATS_VSS_TDGSTCLUST<32|64>
124 .Fo stats_tpl_add_voistats
125 .Fa "uint32_t tpl_id"
127 .Fa "const char *voi_name"
128 .Fa "enum vsd_dtype voi_dtype"
130 .Fa "struct voistatspec *vss"
133 .Ss Stats Blob Data Gathering Functions
135 .Fo stats_voi_update_<abs|rel>_<dtype>
136 .Fa "struct statsblob *sb"
140 .Ss Stats Blob Utility Functions
141 .Ft struct statsblob *
143 .Fa "uint32_t tpl_id"
148 .Fa "struct statsblob *sb"
149 .Fa "uint32_t tpl_id"
154 .Fa "struct statsblob **dst"
155 .Fa "size_t dstmaxsz"
156 .Fa "struct statsblob *src"
160 .Fo stats_blob_destroy
161 .Fa "struct statsblob *sb"
164 .Fo stats_voistat_fetch_dptr
165 .Fa "struct statsblob *sb"
167 .Fa "enum voi_stype stype"
168 .Fa "enum vsd_dtype *retdtype"
169 .Fa "struct voistatdata **retvsd"
170 .Fa "size_t *retvsdsz"
173 .Fo stats_voistat_fetch_<dtype>
174 .Fa "struct statsblob *sb"
176 .Fa "enum voi_stype stype"
180 .Fo stats_blob_snapshot
181 .Fa "struct statsblob **dst"
182 .Fa "size_t dstmaxsz"
183 .Fa "struct statsblob *src"
188 .Fa "struct statsblob *sb"
189 .Fa "struct sbuf *buf"
190 .Fa "enum sb_str_fmt fmt"
194 .Fo stats_voistatdata_tostr
195 .Fa "const struct voistatdata *vsd"
196 .Fa "enum vsd_dtype dtype"
197 .Fa "enum sb_str_fmt fmt"
198 .Fa "struct sbuf *buf"
202 .Fn "\*(lp*stats_blob_visitcb_t\*(rp" "struct sb_visit *sbv" "void *usrctx"
205 .Fa "struct statsblob *sb"
206 .Fa "stats_blob_visitcb_t func"
212 framework facilitates real-time kernel and user space statistics gathering.
213 The framework is built around the
215 an object embedded within a contiguous memory allocation that is mostly opaque
216 to consumers and stores all required state.
219 object can itself be embedded within other objects either directly or indirectly
222 Objects or subsystems for which statistics are to be gathered are initialized
225 which acts as the blueprint for an arbitrary set of
226 Variables Of Interest (VOIs) and their associated statistics.
227 Each template defines a schema plus associated metadata, which are kept separate
228 to minimize the memory footprint of blobs.
230 Data gathering hook functions added at appropriate locations within the code
231 base of interest feed VOI data into the framework for processing.
237 header and opaque internal blob structure per the following diagram:
238 .Bd -literal -offset indent
239 ---------------------------------------------------------
241 | statsblob | opaque[] |
242 ---------------------------------------------------------
245 The publicly visible 8-byte header is defined as:
246 .Bd -literal -offset indent
258 specifies which API version the blob's
261 .Pq Dv STATS_ABI_V1 is the only version currently defined .
263 specifies the endianness of the blob's fields
270 for unknown endianness
273 specifies the size of the blob, while
275 specifies the size of the underlying memory allocation in which the
281 default to units of bytes, unless a flag is set in
283 that dictates otherwise.
285 Templates are constructed by associating arbitrary VOI IDs with a set of
286 statistics, where each statistic is specified using a
287 .Vt struct voistatspec
288 per the definition below:
289 .Bd -literal -offset indent
292 struct vss_hlpr_info *hlprinfo;
293 struct voistatdata *iv;
296 enum vsd_dtype vs_dtype : 8;
297 enum voi_stype stype : 8;
301 It is generally expected that consumers will not work with
302 .Vt struct voistatspec
303 directly, and instead use the
309 framework offers the following statistics for association with VOIs:
310 .Bl -tag -width ".Dv VS_STYPE_TDGST"
312 The sum of VOI values.
314 The maximum VOI value.
316 The minimum VOI value.
318 A static bucket histogram of VOI values, including a count of
319 .Dq out-of-band/bucket Dc
320 values which did not match any bucket.
321 Histograms can be specified as
322 .Dq Em C Ns ontinuous Em R Ns ange Dc
324 .Dq Em D Ns iscrete Em R Ns ange Dc
327 .Dq Em D Ns iscrete Em V Ns alue Dc
329 with 32 or 64 bit bucket counters, depending on the VOI semantics.
330 .It Dv VS_STYPE_TDGST
331 A dynamic bucket histogram of VOI values based on the t-digest method
332 .Po refer to the t-digest paper in the
339 .Dq visitor software design pattern Ns
340 -like scheme is employed to facilitate iterating over a blob's data without
341 concern for the blob's structure.
342 The data provided to visitor callback functions is encapsulated in
344 per the definition below:
345 .Bd -literal -offset indent
347 struct voistatdata *vs_data;
352 enum vsd_dtype voi_dtype : 8;
353 enum vsd_dtype vs_dtype : 8;
360 .Fn stats_tpl_sample_rates
362 .Fn stats_tpl_sample_rollthedice
364 .Vt struct stats_tpl_sample_rate
365 to encapsulate per-template sample rate information per the definition below:
366 .Bd -literal -offset indent
367 struct stats_tpl_sample_rate {
369 uint32_t tpl_sample_pct;
375 member holds the template's slot ID obtained from
378 .Fn stats_tpl_fetch_allocid .
381 member holds the template's sample rate as an integer percentage in the range
385 .Vt stats_tpl_sr_cb_t
386 conformant function pointer that is required as the
389 .Fn stats_tpl_sample_rates
391 .Bd -literal -offset indent
392 enum stats_tpl_sr_cb_action {
398 typedef int (*stats_tpl_sr_cb_t)(enum stats_tpl_sr_cb_action action,
399 struct stats_tpl_sample_rate **rates, int *nrates, void *ctx);
402 It is required that a conformant function:
405 Return an appropriate
407 on error, otherwise 0.
410 .Qq action == TPL_SR_*_GET ,
411 return the subsystem's rates list ptr and count, locked or unlocked as
415 .Qq action == TPL_SR_RUNLOCK ,
416 unlock the subsystem's rates list ptr and count.
418 .Qq action == TPL_SR_RLOCKED_GET
422 .Qq action == TPL_SR_PUT ,
423 update the subsystem's rates list ptr and count to the sysctl processed values
424 and return the inactive list details in
428 for garbage collection by
429 .Fn stats_tpl_sample_rates .
432 Where templates need to be referenced via textual means, for example via a MIB
433 variable, the following string based template spec formats can be used:
439 .Qq TCP_DEFAULT Qc Ns
451 The first form is the normative spec format generated by the framework, while
452 the second and third forms are convenience formats primarily for user input.
453 The use of inverted commas around the template name is optional.
457 framework exposes the following framework-specific variables in the
462 .Bl -tag -width "templates"
464 Read-only CSV list of registered templates in normative template spec form.
466 .Ss Template Management Functions
469 function allocates a new template with the specified unique name and returns its
470 runtime-stable template slot ID for use with other API functions.
473 argument is currently unused.
476 .Fn stats_tpl_fetch_allocid
477 function returns the runtime-stable template slot ID of any registered template
478 matching the specified name and hash.
482 function returns the pointer to the registered template object at the specified
486 .Fn stats_tpl_id2name
487 function returns the name of the registered template object at the specified
491 .Fn stats_tpl_sample_rates
492 function provides a generic handler for template sample rates management and
496 Subsystems can use this function to create a subsystem-specific
498 MIB variable that manages and reports subsystem-specific template sampling
500 Subsystems must supply a
501 .Vt stats_tpl_sr_cb_t
502 conformant function pointer as the sysctl's
504 which is a callback used to interact with the subsystem's stats template sample
506 Subsystems can optionally specify the sysctl's
508 as non-zero, which causes a zero-initialized allocation of arg2-sized contextual
509 memory to be heap-allocated and passed in to all subsystem callbacks made during
511 .Fn stats_tpl_sample_rates .
514 .Fn stats_tpl_sample_rollthedice
515 function makes a weighted random template selection from the supplied array of
516 template sampling rates.
517 The cumulative percentage of all sampling rates should not exceed 100.
518 If no seed is supplied, a PRNG is used to generate a true random number so that
519 every selection is independent.
520 If a seed is supplied, selection will be made randomly across different seeds, but
521 deterministically given the same seed.
524 .Fn stats_tpl_add_voistats
525 function is used to add a VOI and associated set of statistics to the registered
526 template object at the specified template slot ID.
527 The set of statistics is passed as an array of
528 .Vt struct voistatspec
529 which can be initialized using the
531 helper macros or manually for non-standard use cases.
536 count of array elements can be determined by passing
543 flag can be passed to configure the VOI for use with
544 .Fn stats_voi_update_rel_<dtype> ,
545 which entails maintaining an extra 8 bytes of state in the blob at each update.
546 .Ss Data Gathering Functions
548 .Fn stats_voi_update_abs_<dtype>
550 .Fn stats_voi_update_rel_<dtype>
551 functions both update all the statistics associated with the VOI identified by
557 as an absolute value, whereas the
561 as a value relative to that of the previous update function call, by adding it
562 to the previous value and using the result for the update.
563 Relative updates are only possible for VOIs that were added to the template with
567 .Fn stats_tpl_add_voistats .
568 .Ss Utility Functions
571 function allocates and initializes a new blob based on the registered template
572 object at the specified template slot ID.
576 function initializes a new blob in an existing memory allocation based on the
577 registered template object at the specified template slot ID.
581 function duplicates the
591 .Dv SB_CLONE_ALLOCDST
592 flag can be passed to instruct the function to allocate a new blob of
593 appropriate size into which to clone
595 storing the new pointer in
598 .Dv SB_CLONE_USRDSTNOFAULT
601 flags can be set to respectively signal that
602 .Xr copyout_nofault 9
605 should be used because
607 is a user space address.
610 .Fn stats_blob_snapshot
615 and then performs any additional functions required to produce a coherent
617 The flags interpreted by
620 .Fn stats_blob_snapshot .
623 flag can be used to effect a reset of the
625 blob's statistics after a snapshot is successfully taken.
628 .Fn stats_blob_destroy
629 function destroys a blob previously created with
630 .Fn stats_blob_alloc ,
633 .Fn stats_blob_snapshot .
637 function allows the caller to iterate over the contents of a blob.
638 The callback function
640 is called for every VOI and statistic in the blob, passing a
642 and the user context argument
644 to the callback function.
647 passed to the callback function may have one or more of the following flags set
650 struct member to provide useful metadata about the iteration:
653 .Dv SB_IT_FIRST_VOI ,
655 .Dv SB_IT_FIRST_VOISTAT ,
656 .Dv SB_IT_LAST_VOISTAT ,
659 .Dv SB_IT_NULLVOISTAT .
660 Returning a non-zero value from the callback function terminates the iteration.
664 renders a string representation of a blob into the
667 Currently supported render formats are
668 .Dv SB_STRFMT_FREEFORM
673 flag can be passed to render version specific opaque implementation detail for
674 debugging or string-to-binary blob reconstruction purposes.
677 flag can be passed to render template metadata into the string representation,
678 using the blob's template hash to lookup the corresponding template.
681 .Fn stats_voistatdata_tostr
682 renders a string representation of an individual statistic's data into the
685 The same render formats supported by the
687 function can be specified, and the
689 boolean has the same meaning as the
694 .Fn stats_voistat_fetch_dptr
695 function returns an internal blob pointer to the specified
697 statistic data for the VOI
700 .Fn stats_voistat_fetch_<dtype>
701 functions are convenience wrappers around
702 .Fn stats_voistat_fetch_dptr
703 to perform the extraction for simple data types.
704 .Sh IMPLEMENTATION NOTES
705 The following notes apply to STATS_ABI_V1 format statsblobs.
706 .Ss Space-Time Complexity
707 Blobs are laid out as three distinct memory regions following the header:
708 .Bd -literal -offset indent
709 ------------------------------------------------------
710 | struct | struct | struct | struct |
711 | statsblobv1 | voi [] | voistat [] | voistatdata [] |
712 ------------------------------------------------------
715 Blobs store VOI and statistic blob state
723 in sparse arrays, using the
728 This allows O(1) access to any voi/voistat pair in the blob, at the expense of
729 8 bytes of wasted memory per vacant slot for templates which do not specify
730 contiguously numbered VOIs and/or statistic types.
731 Data storage for statistics is only allocated for non-vacant slot pairs.
733 To provide a concrete example, a blob with the following specification:
736 Two VOIs; ID 0 and 2; added to the template in that order
738 VOI 0 is of data type
742 to enable support for relative updates using
743 .Fn stats_voi_update_rel_<dtype> ,
746 statistic associated with it.
748 VOI 2 is of data type
754 statistics associated with it.
757 would have the following memory layout:
759 --------------------------------------
760 | header | struct statsblobv1, 32 bytes
761 |------------------------------------|
762 | voi[0] | struct voi, 8 bytes
763 | voi[1] (vacant) | struct voi, 8 bytes
764 | voi[2] | struct voi, 8 bytes
765 |------------------------------------|
766 | voi[2]voistat[VOISTATE] (vacant) | struct voistat, 8 bytes
767 | voi[2]voistat[SUM] | struct voistat, 8 bytes
768 | voi[2]voistat[MAX] | struct voistat, 8 bytes
769 | voi[0]voistat[VOISTATE] | struct voistat, 8 bytes
770 | voi[0]voistat[SUM] (vacant) | struct voistat, 8 bytes
771 | voi[0]voistat[MAX] (vacant) | struct voistat, 8 bytes
772 | voi[0]voistat[MIN] | struct voistat, 8 bytes
773 |------------------------------------|
774 | voi[2]voistat[SUM]voistatdata | struct voistatdata_int32, 4 bytes
775 | voi[2]voistat[MAX]voistatdata | struct voistatdata_int32, 4 bytes
776 | voi[0]voistat[VOISTATE]voistatdata | struct voistatdata_numeric, 8 bytes
777 | voi[0]voistat[MIN]voistatdata | struct voistatdata_int64, 8 bytes
778 --------------------------------------
782 When rendered to string format using
783 .Fn stats_blob_tostr ,
785 .Dv SB_STRFMT_FREEFORM
789 flag, the rendered output is:
791 struct statsblobv1@0x8016250a0, abi=1, endian=1, maxsz=136, cursz=136, \\
792 created=6294158585626144, lastrst=6294158585626144, flags=0x0000, \\
793 stats_off=56, statsdata_off=112, tplhash=2994056564
794 vois[0]: id=0, name="", flags=0x0001, dtype=INT_S64, voistatmaxid=3, \\
796 vois[0]stat[0]: stype=VOISTATE, flags=0x0000, dtype=VOISTATE, \\
799 vois[0]stat[1]: stype=-1
800 vois[0]stat[2]: stype=-1
801 vois[0]stat[3]: stype=MIN, flags=0x0000, dtype=INT_S64, \\
803 voistatdata: 9223372036854775807
805 vois[2]: id=2, name="", flags=0x0000, dtype=INT_U32, voistatmaxid=2, \\
807 vois[2]stat[0]: stype=-1
808 vois[2]stat[1]: stype=SUM, flags=0x0000, dtype=INT_U32, dsz=4, \\
811 vois[2]stat[2]: stype=MAX, flags=0x0000, dtype=INT_U32, dsz=4, \\
818 present in the rendered output above indicates a manual line break inserted to
819 keep the man page within 80 columns and is not part of the actual output.
823 framework does not provide any concurrency protection at the individual blob
824 level, instead requiring that consumers guarantee mutual exclusion when calling
825 API functions that reference a non-template blob.
827 The list of templates is protected with a
831 rw lock in user space to support concurrency between template management and
832 blob initialization operations.
835 returns a runtime-stable template slot ID on success, or a negative errno on
837 -EINVAL is returned if any problems are detected with the arguments.
838 -EEXIST is returned if an existing template is registered with the same name.
839 -ENOMEM is returned if a required memory allocation fails.
841 .Fn stats_tpl_fetch_allocid
842 returns a runtime-stable template slot ID, or negative errno on failure.
843 -ESRCH is returned if no registered template matches the specified name and/or
847 returns 0 on success, or ENOENT if an invalid
851 .Fn stats_tpl_id2name
852 returns 0 on success, or an errno on failure.
853 EOVERFLOW is returned if the length of
857 is too short to hold the template's name.
858 ENOENT is returned if an invalid
862 .Fn stats_tpl_sample_rollthedice
863 returns a valid template slot id selected from
865 or -1 if a NULL selection was made, that is no stats collection this roll.
867 .Fn stats_tpl_add_voistats
868 return 0 on success, or an errno on failure.
869 EINVAL is returned if any problems are detected with the arguments.
870 EFBIG is returned if the resulting blob would have exceeded the maximum size.
871 EOPNOTSUPP is returned if an attempt is made to add more VOI stats to a
872 previously configured VOI.
873 ENOMEM is returned if a required memory allocation fails.
875 .Fn stats_voi_update_abs_<dtype>
877 .Fn stats_voi_update_rel_<dtype>
878 return 0 on success, or EINVAL if any problems are detected with the arguments.
881 returns 0 on success, or an errno on failure.
882 EINVAL is returned if any problems are detected with the arguments.
883 EOVERFLOW is returned if the template blob's
887 of the blob being initialized.
890 returns a pointer to a newly allocated and initialized blob based on the
891 specified template with slot ID
893 or NULL if the memory allocation failed.
897 .Fn stats_blob_snapshot
898 return 0 on success, or an errno on failure.
899 EINVAL is returned if any problems are detected with the arguments.
900 ENOMEM is returned if the SB_CLONE_ALLOCDST flag was specified and the memory
904 EOVERFLOW is returned if the src blob's
913 returns 0 on success, or EINVAL if any problems are detected with the arguments.
917 .Fn stats_voistatdata_tostr
918 return 0 on success, or an errno on failure.
919 EINVAL is returned if any problems are detected with the arguments, otherwise
920 any error returned by
926 .Fn stats_voistat_fetch_dptr
927 returns 0 on success, or EINVAL if any problems are detected with the arguments.
929 .Fn stats_voistat_fetch_<dtype>
930 returns 0 on success, or an errno on failure.
931 EINVAL is returned if any problems are detected with the arguments.
932 EFTYPE is returned if the requested data type does not match the blob's data
933 type for the specified
946 .%T "Computing Extremely Accurate Quantiles Using t-digests"
947 .%U "https://github.com/tdunning/t-digest/raw/master/docs/t-digest-paper/histo.pdf"
952 framework first appeared in
958 framework and this manual page were written by
959 .An Lawrence Stewart Aq lstewart@FreeBSD.org
960 and sponsored by Netflix, Inc.
962 Granularity of timing-dependent network statistics, in particular TCP_RTT,
966 To minimize the measurement error avoid using HZ lower than 1000.