2 .\" Copyright (c) 2010 The FreeBSD Foundation
3 .\" All rights reserved.
5 .\" Portions of this software were developed at the Centre for Advanced
6 .\" Internet Architectures, Swinburne University of Technology, Melbourne,
7 .\" Australia by Lawrence Stewart under sponsorship from the FreeBSD
10 .\" Redistribution and use in source and binary forms, with or without
11 .\" modification, are permitted provided that the following conditions
13 .\" 1. Redistributions of source code must retain the above copyright
14 .\" notice, this list of conditions, and the following disclaimer,
15 .\" without modification, immediately at the beginning of the file.
16 .\" 2. The name of the author may not be used to endorse or promote products
17 .\" derived from this software without specific prior written permission.
19 .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
20 .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
21 .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
22 .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR
23 .\" ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
24 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
25 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
26 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
27 .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
28 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
38 .Nd Statistical Information For TCP Research
42 as a module at run-time, run the following command as root:
43 .Bd -literal -offset indent
47 Alternatively, to load
49 as a module at boot time, add the following line into the
52 .Bd -literal -offset indent
65 kernel module logs a range of statistics on active TCP connections to
67 It provides the ability to make highly granular measurements of TCP connection
68 state, aimed at system administrators, developers and researchers.
69 .Ss Compile-time Configuration
70 The default operation of
72 is to capture IPv4 TCP/IP packets.
74 can be configured to support IPv4 and IPv6 by uncommenting:
75 .Bd -literal -offset indent
80 .Aq sys/modules/siftr/Makefile
83 In the IPv4-only (default) mode, standard dotted decimal notation (e.g.
84 "136.186.229.95") is used to format IPv4 addresses for logging.
85 In IPv6 mode, standard dotted decimal notation is used to format IPv4 addresses,
86 and standard colon-separated hex notation (see RFC 4291) is used to format IPv6
87 addresses for logging. Note that SIFTR uses uncompressed notation to format IPv6
89 For example, the address "fe80::20f:feff:fea2:531b" would be logged as
90 "fe80:0:0:0:20f:feff:fea2:531b".
91 .Ss Run-time Configuration
95 interface to export its configuration variables to user-space.
96 The following variables are available:
97 .Bl -tag -offset indent -width Va
98 .It Va net.inet.siftr.enabled
99 controls whether the module performs its
101 By default, the value is set to 0, which means the module
102 will not be taking any measurements.
103 Having the module loaded with
104 .Va net.inet.siftr.enabled
105 set to 0 will have no impact on the performance of the network stack, as the
106 packet filtering hooks are only inserted when
107 .Va net.inet.siftr.enabled
110 .Bl -tag -offset indent -width Va
111 .It Va net.inet.siftr.ppl
112 controls how many inbound/outbound packets for a given TCP connection will cause
113 a log message to be generated for the connection.
114 By default, the value is set to 1, which means the module will log a message for
115 every packet of every TCP connection.
116 The value can be set to any integer in the range [1,2^32], and can be changed at
117 any time, even while the module is enabled.
119 .Bl -tag -offset indent -width Va
120 .It Va net.inet.siftr.logfile
121 controls the path to the file that the module writes its log messages to.
122 By default, the file /var/log/siftr.log is used.
123 The path can be changed at any time, even while the module is enabled.
125 .Bl -tag -offset indent -width Va
126 .It Va net.inet.siftr.genhashes
127 controls whether a hash is generated for each TCP packet seen by
129 By default, the value is set to 0, which means no hashes are generated.
130 The hashes are useful to correlate which TCP packet triggered the generation of
131 a particular log message, but calculating them adds additional computational
132 overhead into the fast path.
137 log file will contain 3 different types of log message.
138 All messages are written in plain ASCII text.
142 present in the example log messages in this section indicates a
143 line continuation and is not part of the actual log message.
145 The first type of log message is written to the file when the module is
146 enabled and starts collecting data from the running kernel. The text below
147 shows an example module enable log. The fields are tab delimited key-value
148 pairs which describe some basic information about the system.
149 .Bd -literal -offset indent
150 enable_time_secs=1238556193 enable_time_usecs=462104 \\
151 siftrver=1.2.2 hz=1000 tcp_rtt_scale=32 \\
152 sysname=FreeBSD sysver=604000 ipmode=4
155 Field descriptions are as follows:
156 .Bl -tag -offset indent -width Va
157 .It Va enable_time_secs
158 time at which the module was enabled, in seconds since the UNIX epoch.
160 .Bl -tag -offset indent -width Va
161 .It Va enable_time_usecs
162 time at which the module was enabled, in microseconds since enable_time_secs.
164 .Bl -tag -offset indent -width Va
169 .Bl -tag -offset indent -width Va
171 tick rate of the kernel in ticks per second.
173 .Bl -tag -offset indent -width Va
175 smoothed RTT estimate scaling factor.
177 .Bl -tag -offset indent -width Va
179 operating system name.
181 .Bl -tag -offset indent -width Va
183 operating system version.
185 .Bl -tag -offset indent -width Va
187 IP mode as defined at compile time.
188 An ipmode of "4" means IPv6 is not supported and IP addresses are logged in
189 regular dotted quad format.
190 An ipmode of "6" means IPv6 is supported, and IP addresses are logged in dotted
191 quad or hex format, as described in the
192 .Qq Compile-time Configuration
196 The second type of log message is written to the file when a data log message
198 The text below shows an example data log triggered by an IPv4
200 The data is CSV formatted.
201 .Bd -literal -offset indent
202 o,0xbec491a5,1238556193.463551,172.16.7.28,22,172.16.2.5,55931, \\
203 1073725440,172312,6144,66560,66608,8,1,4,1448,936,1,996,255, \\
204 33304,208,66608,0,208,0
207 Field descriptions are as follows:
208 .Bl -tag -offset indent -width Va
210 Direction of packet that triggered the log message.
217 .Bl -tag -offset indent -width Va
219 Hash of the packet that triggered the log message.
221 .Bl -tag -offset indent -width Va
223 Time at which the packet that triggered the log message was processed by
226 hook function, in seconds and microseconds since the UNIX epoch.
228 .Bl -tag -offset indent -width Va
230 The IPv4 or IPv6 address of the local host, in dotted quad (IPv4 packet)
231 or colon-separated hex (IPv6 packet) notation.
233 .Bl -tag -offset indent -width Va
235 The TCP port that the local host is communicating via.
237 .Bl -tag -offset indent -width Va
239 The IPv4 or IPv6 address of the foreign host, in dotted quad (IPv4 packet)
240 or colon-separated hex (IPv6 packet) notation.
242 .Bl -tag -offset indent -width Va
244 The TCP port that the foreign host is communicating via.
246 .Bl -tag -offset indent -width Va
248 The slow start threshold for the flow, in bytes.
250 .Bl -tag -offset indent -width Va
252 The current congestion window for the flow, in bytes.
254 .Bl -tag -offset indent -width Va
256 The current bandwidth-controlled window for the flow, in bytes.
258 .Bl -tag -offset indent -width Va
260 The current sending window for the flow, in bytes.
261 The post scaled value is reported, except during the initial handshake (first
262 few packets), during which time the unscaled value is reported.
264 .Bl -tag -offset indent -width Va
266 The current receive window for the flow, in bytes.
267 The post scaled value is always reported.
269 .Bl -tag -offset indent -width Va
271 The current window scaling factor for the sending window.
273 .Bl -tag -offset indent -width Va
275 The current window scaling factor for the receiving window.
277 .Bl -tag -offset indent -width Va
279 The current state of the TCP finite state machine, as defined
281 .Aq Pa netinet/tcp_fsm.h .
283 .Bl -tag -offset indent -width Va
285 The maximum segment size for the flow, in bytes.
287 .Bl -tag -offset indent -width Va
289 The current smoothed RTT estimate for the flow, in units of TCP_RTT_SCALE * HZ,
290 where TCP_RTT_SCALE is a define found in tcp_var.h, and HZ is the kernel's tick
292 Divide by TCP_RTT_SCALE * HZ to get the RTT in secs. TCP_RTT_SCALE and HZ are
293 reported in the enable log message.
295 .Bl -tag -offset indent -width Va
297 SACK enabled indicator. 1 if SACK enabled, 0 otherwise.
299 .Bl -tag -offset indent -width Va
301 The current state of the TCP flags for the flow.
303 .Aq Pa netinet/tcp_var.h
304 for information about the various flags.
306 .Bl -tag -offset indent -width Va
308 The current retransmission timeout length for the flow, in units of HZ, where HZ
309 is the kernel's tick timer.
310 Divide by HZ to get the timeout length in seconds. HZ is reported in the
313 .Bl -tag -offset indent -width Va
315 The current size of the socket send buffer in bytes.
317 .Bl -tag -offset indent -width Va
319 The current number of bytes in the socket send buffer.
321 .Bl -tag -offset indent -width Va
323 The current size of the socket receive buffer in bytes.
325 .Bl -tag -offset indent -width Va
327 The current number of bytes in the socket receive buffer.
329 .Bl -tag -offset indent -width Va
331 The current number of unacknowledged bytes in-flight.
332 Bytes acknowledged via SACK are not excluded from this count.
334 .Bl -tag -offset indent -width Va
336 The current number of segments in the reassembly queue.
339 The third type of log message is written to the file when the module is disabled
340 and ceases collecting data from the running kernel.
341 The text below shows an example module disable log.
342 The fields are tab delimited key-value pairs which provide statistics about
343 operations since the module was most recently enabled.
344 .Bd -literal -offset indent
345 disable_time_secs=1238556197 disable_time_usecs=933607 \\
346 num_inbound_tcp_pkts=356 num_outbound_tcp_pkts=627 \\
347 total_tcp_pkts=983 num_inbound_skipped_pkts_malloc=0 \\
348 num_outbound_skipped_pkts_malloc=0 num_inbound_skipped_pkts_mtx=0 \\
349 num_outbound_skipped_pkts_mtx=0 num_inbound_skipped_pkts_tcb=0 \\
350 num_outbound_skipped_pkts_tcb=0 num_inbound_skipped_pkts_icb=0 \\
351 num_outbound_skipped_pkts_icb=0 total_skipped_tcp_pkts=0 \\
352 flow_list=172.16.7.28;22-172.16.2.5;55931,
355 Field descriptions are as follows:
356 .Bl -tag -offset indent -width Va
357 .It Va disable_time_secs
358 Time at which the module was disabled, in seconds since the UNIX epoch.
360 .Bl -tag -offset indent -width Va
361 .It Va disable_time_usecs
362 Time at which the module was disabled, in microseconds since disable_time_secs.
364 .Bl -tag -offset indent -width Va
365 .It Va num_inbound_tcp_pkts
366 Number of TCP packets that traversed up the network stack.
367 This only includes inbound TCP packets during the periods when
371 .Bl -tag -offset indent -width Va
372 .It Va num_outbound_tcp_pkts
373 Number of TCP packets that traversed down the network stack.
374 This only includes outbound TCP packets during the periods when
378 .Bl -tag -offset indent -width Va
379 .It Va total_tcp_pkts
380 The summation of num_inbound_tcp_pkts and num_outbound_tcp_pkts.
382 .Bl -tag -offset indent -width Va
383 .It Va num_inbound_skipped_pkts_malloc
384 Number of inbound packets that were not processed because of failed malloc() calls.
386 .Bl -tag -offset indent -width Va
387 .It Va num_outbound_skipped_pkts_malloc
388 Number of outbound packets that were not processed because of failed malloc() calls.
390 .Bl -tag -offset indent -width Va
391 .It Va num_inbound_skipped_pkts_mtx
392 Number of inbound packets that were not processed because of failure to add the
393 packet to the packet processing queue.
395 .Bl -tag -offset indent -width Va
396 .It Va num_outbound_skipped_pkts_mtx
397 Number of outbound packets that were not processed because of failure to add the
398 packet to the packet processing queue.
400 .Bl -tag -offset indent -width Va
401 .It Va num_inbound_skipped_pkts_tcb
402 Number of inbound packets that were not processed because of failure to find the
403 TCP control block associated with the packet.
405 .Bl -tag -offset indent -width Va
406 .It Va num_outbound_skipped_pkts_tcb
407 Number of outbound packets that were not processed because of failure to find
408 the TCP control block associated with the packet.
410 .Bl -tag -offset indent -width Va
411 .It Va num_inbound_skipped_pkts_icb
412 Number of inbound packets that were not processed because of failure to find the
413 IP control block associated with the packet.
415 .Bl -tag -offset indent -width Va
416 .It Va num_outbound_skipped_pkts_icb
417 Number of outbound packets that were not processed because of failure to find
418 the IP control block associated with the packet.
420 .Bl -tag -offset indent -width Va
421 .It Va total_skipped_tcp_pkts
422 The summation of all skipped packet counters.
424 .Bl -tag -offset indent -width Va
426 A CSV list of TCP flows that triggered data log messages to be generated since
427 the module was loaded.
428 Each flow entry in the CSV list is
430 .Qq local_ip;local_port-foreign_ip;foreign_port .
431 If there are no entries in the list (i.e., no data log messages were generated),
432 the value will be blank.
433 If there is at least one entry in the list, a trailing comma will always be
437 The total number of data log messages found in the log file for a module
438 enable/disable cycle should equate to total_tcp_pkts - total_skipped_tcp_pkts.
439 .Sh IMPLEMENTATION NOTES
441 hooks into the network stack using the
444 In its current incarnation, it hooks into the AF_INET/AF_INET6 (IPv4/IPv6)
446 filtering points, which means it sees packets at the IP layer of the network
448 This means that TCP packets inbound to the stack are intercepted before
449 they have been processed by the TCP layer.
450 Packets outbound from the stack are intercepted after they have been processed
453 The diagram below illustrates how
455 inserts itself into the stack.
456 .Bd -literal -offset indent
457 ----------------------------------
459 ----------------------------------
465 ----------------------------------
474 ________| |__________
477 IPv{4/6} in IPv{4/6} out
478 ----------------------------------
482 Layer 2 in Layer 2 out
483 ----------------------------------
485 ----------------------------------
491 interface to manage writing data to disk.
493 At first glance, you might mistakenly think that
495 extracts information from
496 individual TCP packets.
497 This is not the case.
499 uses TCP packet events (inbound and outbound) for each TCP flow originating from
500 the system to trigger a dump of the state of the TCP control block for that
502 With the PPL set to 1, we are in effect sampling each TCP flow's control block
503 state as frequently as flow packets enter/leave the system.
504 For example, setting PPL to 2 halves the sampling rate i.e., every second flow
505 packet (inbound OR outbound) causes a dump of the control block state.
507 The distinction between interrogating individual packets versus interrogating the
508 control block is important, because
510 does not remove the need for packet capturing tools like
513 allows you to correlate and observe the cause-and-affect relationship between
514 what you see on the wire (captured using a tool like
516 and changes in the TCP control block corresponding to the flow of interest.
517 It is therefore useful to use
521 to gather the necessary data to piece together the complete picture.
522 Use of either tool on its own will not be able to provide all of the necessary
525 As a result of needing to interrogate the TCP control block, certain packets
526 during the lifecycle of a connection are unable to trigger a
529 The initial handshake takes place without the existence of a control block and
530 the final ACK is exchanged when the connection is in the TIMEWAIT state.
533 was designed to minimise the delay introduced to packets traversing the network
535 This design called for a highly optimised and minimal hook function that
536 extracted the minimal details necessary whilst holding the packet up, and
537 passing these details to another thread for actual processing and logging.
539 This multithreaded design does introduce some contention issues when accessing
540 the data structure shared between the threads of operation.
541 When the hook function tries to place details in the structure, it must first
542 acquire an exclusive lock.
543 Likewise, when the processing thread tries to read details from the structure,
544 it must also acquire an exclusive lock to do so.
545 If one thread holds the lock, the other must wait before it can obtain it.
546 This does introduce some additional bounded delay into the kernel's packet
547 processing code path.
549 In some cases (e.g., low memory, connection termination), TCP packets that enter
553 hook function will not trigger a log message to be generated.
555 refers to this outcome as a
559 always ensures that packets are allowed to continue through the stack, even if
560 they could not successfully trigger a data log message.
562 will therefore not introduce any packet loss for TCP/IP packets traversing the
564 .Ss Important Behaviours
565 The behaviour of a log file path change whilst the module is enabled is as
569 Attempt to open the new file path for writing.
570 If this fails, the path change will fail and the existing path will continue to
573 Assuming the new path is valid and opened successfully:
576 Flush all pending log messages to the old file path.
578 Close the old file path.
580 Switch the active log file pointer to point at the new file path.
582 Commence logging to the new file.
586 During the time between the flush of pending log messages to the old file and
587 commencing logging to the new file, new log messages will still be generated and
589 As soon as the new file path is ready for writing, the accumulated log messages
590 will be written out to the file.
592 To enable the module's operations, run the following command as root:
593 sysctl net.inet.siftr.enabled=1
595 To change the granularity of log messages such that 1 log message is
596 generated for every 10 TCP packets per connection, run the following
598 sysctl net.inet.siftr.ppl=10
600 To change the log file location to /tmp/siftr.log, run the following
602 sysctl net.inet.siftr.logfile=/tmp/siftr.log
610 Development of this software was made possible in part by grants from the
611 Cisco University Research Program Fund at Community Foundation Silicon Valley,
612 and the FreeBSD Foundation.
621 was first released in 2007 by Lawrence Stewart and James Healy whilst working on
622 the NewTCP research project at Swinburne University of Technology's Centre for
623 Advanced Internet Architectures, Melbourne, Australia, which was made possible
624 in part by a grant from the Cisco University Research Program Fund at Community
625 Foundation Silicon Valley.
626 More details are available at:
628 http://caia.swin.edu.au/urp/newtcp/
632 v1.2.x was sponsored by the FreeBSD Foundation as part of
634 .Qq Enhancing the FreeBSD TCP Implementation
636 More details are available at:
638 http://www.freebsdfoundation.org/
640 http://caia.swin.edu.au/freebsd/etcp09/
645 .An Lawrence Stewart Aq lstewart@FreeBSD.org
647 .An James Healy Aq jimmy@deefa.com .
649 This manual page was written by
650 .An Lawrence Stewart Aq lstewart@FreeBSD.org .
652 Current known limitations and any relevant workarounds are outlined below:
655 The internal queue used to pass information between the threads of operation is
659 to cope with bursty network traffic, but sustained high packet-per-second
660 traffic can cause exhaustion of kernel memory if the processing thread cannot
661 keep up with the packet rate.
665 on a machine that is also running other modules utilising the
671 the order in which you load the modules is important.
672 You should kldload the other modules first, as this will ensure TCP packets
673 undergo any necessary manipulations before
678 There is a known, harmless lock order reversal warning between the
680 mutex and tcbinfo TCP lock reported by
684 is enabled in a kernel compiled with
688 There is no way to filter which TCP flows you wish to capture data for.
689 Post processing is required to separate out data belonging to particular flows
692 The module does not detect deletion of the log file path.
693 New log messages will simply be lost if the log file being used by
695 is deleted whilst the module is set to use the file.
696 Switching to a new log file using the
697 .Em net.inet.siftr.logfile
698 variable will create the new file and allow log messages to begin being written
700 The new log file path must differ from the path to the deleted file.
702 The hash table used within the code is sized to hold 65536 flows. This is not a
703 hard limit, because chaining is used to handle collisions within the hash table
705 However, we suspect (based on analogies with other hash table performance data)
706 that the hash table look up performance (and therefore the module's packet
707 processing performance) will degrade in an exponential manner as the number of
708 unique flows handled in a module enable/disable cycle approaches and surpasses
711 There is no garbage collection performed on the flow hash table.
712 The only way currently to flush it is to disable
715 The PPL variable applies to packets that make it into the processing thread,
716 not total packets received in the hook function.
717 Packets are skipped before the PPL variable is applied, which means there may be
718 a slight discrepancy in the triggering of log messages.
719 For example, if PPL was set to 10, and the 8th packet since the last log message
720 is skipped, the 11th packet will actually trigger the log message to be
722 This is discussed in greater depth in CAIA technical report 070824A.
724 At the time of writing, there was no simple way to hook into the TCP layer
725 to intercept packets.
727 use of IP layer hook points means all IP
728 traffic will be processed by the
731 hook function, which introduces minor, but nonetheless unnecessary packet delay
732 and processing overhead on the system for non-TCP packets as well.
733 Hooking in at the IP layer is also not ideal from the data gathering point of
735 Packets traversing up the stack will be intercepted and cause a log message
736 generation BEFORE they have been processed by the TCP layer, which means we
737 cannot observe the cause-and-affect relationship between inbound events and the
738 corresponding TCP control block as precisely as could be.
741 should intercept packets after they have been processed by the TCP layer i.e.
742 intercept packets coming up the stack after they have been processed by
743 tcp_input(), and intercept packets coming down the stack after they have been
744 processed by tcp_output().
745 The current code still gives satisfactory granularity though, as inbound events
746 tend to trigger outbound events, allowing the cause-and-effect to be observed
747 indirectly by capturing the state on outbound events as well.
753 does not take into account bytes that have been
755 by the receiving host.
757 Packet hash generation does not currently work for IPv6 based TCP packets.
759 Compressed notation is not used for IPv6 address representation.
760 This consumes more bytes than is necessary in log output.