2 .\" Copyright (c) 2010 The FreeBSD Foundation
3 .\" All rights reserved.
5 .\" Portions of this software were developed at the Centre for Advanced
6 .\" Internet Architectures, Swinburne University of Technology, Melbourne,
7 .\" Australia by Lawrence Stewart under sponsorship from the FreeBSD
10 .\" Redistribution and use in source and binary forms, with or without
11 .\" modification, are permitted provided that the following conditions
13 .\" 1. Redistributions of source code must retain the above copyright
14 .\" notice, this list of conditions, and the following disclaimer,
15 .\" without modification, immediately at the beginning of the file.
16 .\" 2. The name of the author may not be used to endorse or promote products
17 .\" derived from this software without specific prior written permission.
19 .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
20 .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
21 .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
22 .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR
23 .\" ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
24 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
25 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
26 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
27 .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
28 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
38 .Nd Statistical Information For TCP Research
42 as a module at run-time, run the following command as root:
43 .Bd -literal -offset indent
47 Alternatively, to load
49 as a module at boot time, add the following line into the
52 .Bd -literal -offset indent
65 kernel module logs a range of statistics on active TCP connections to
67 It provides the ability to make highly granular measurements of TCP connection
68 state, aimed at system administrators, developers and researchers.
69 .Ss Compile-time Configuration
70 The default operation of
72 is to capture IPv4 TCP/IP packets.
74 can be configured to support IPv4 and IPv6 by uncommenting:
75 .Bd -literal -offset indent
80 .Aq sys/modules/siftr/Makefile
83 In the IPv4-only (default) mode, standard dotted decimal notation (e.g.
84 "136.186.229.95") is used to format IPv4 addresses for logging.
85 In IPv6 mode, standard dotted decimal notation is used to format IPv4 addresses,
86 and standard colon-separated hex notation (see RFC 4291) is used to format IPv6
87 addresses for logging.
88 Note that SIFTR uses uncompressed notation to format IPv6 addresses.
89 For example, the address "fe80::20f:feff:fea2:531b" would be logged as
90 "fe80:0:0:0:20f:feff:fea2:531b".
91 .Ss Run-time Configuration
95 interface to export its configuration variables to user-space.
96 The following variables are available:
97 .Bl -tag -offset indent -width Va
98 .It Va net.inet.siftr.enabled
99 controls whether the module performs its
101 By default, the value is set to 0, which means the module
102 will not be taking any measurements.
103 Having the module loaded with
104 .Va net.inet.siftr.enabled
105 set to 0 will have no impact on the performance of the network stack, as the
106 packet filtering hooks are only inserted when
107 .Va net.inet.siftr.enabled
110 .Bl -tag -offset indent -width Va
111 .It Va net.inet.siftr.ppl
112 controls how many inbound/outbound packets for a given TCP connection will cause
113 a log message to be generated for the connection.
114 By default, the value is set to 1, which means the module will log a message for
115 every packet of every TCP connection.
116 The value can be set to any integer in the range [1,2^32], and can be changed at
117 any time, even while the module is enabled.
119 .Bl -tag -offset indent -width Va
120 .It Va net.inet.siftr.logfile
121 controls the path to the file that the module writes its log messages to.
122 By default, the file /var/log/siftr.log is used.
123 The path can be changed at any time, even while the module is enabled.
125 .Bl -tag -offset indent -width Va
126 .It Va net.inet.siftr.genhashes
127 controls whether a hash is generated for each TCP packet seen by
129 By default, the value is set to 0, which means no hashes are generated.
130 The hashes are useful to correlate which TCP packet triggered the generation of
131 a particular log message, but calculating them adds additional computational
132 overhead into the fast path.
134 .Bl -tag -offset indent -width Va
135 .It Va net.inet.siftr.port_filter
136 controls on which source or destination port siftr should capture
138 By default, the value is set to 0, which means all ports are eligible for logging.
139 Set to any other value, only packets where either the source or destination
140 port is equal to this number are logged.
145 log file will contain 3 different types of log message.
146 All messages are written in plain ASCII text.
150 present in the example log messages in this section indicates a
151 line continuation and is not part of the actual log message.
153 The first type of log message is written to the file when the module is
154 enabled and starts collecting data from the running kernel.
155 The text below shows an example module enable log.
156 The fields are tab delimited key-value
157 pairs which describe some basic information about the system.
158 .Bd -literal -offset indent
159 enable_time_secs=1238556193 enable_time_usecs=462104 \\
160 siftrver=1.2.2 hz=1000 tcp_rtt_scale=32 \\
161 sysname=FreeBSD sysver=604000 ipmode=4
164 Field descriptions are as follows:
165 .Bl -tag -offset indent -width Va
166 .It Va enable_time_secs
167 time at which the module was enabled, in seconds since the UNIX epoch.
169 .Bl -tag -offset indent -width Va
170 .It Va enable_time_usecs
171 time at which the module was enabled, in microseconds since enable_time_secs.
173 .Bl -tag -offset indent -width Va
178 .Bl -tag -offset indent -width Va
180 tick rate of the kernel in ticks per second.
182 .Bl -tag -offset indent -width Va
184 smoothed RTT estimate scaling factor.
186 .Bl -tag -offset indent -width Va
188 operating system name.
190 .Bl -tag -offset indent -width Va
192 operating system version.
194 .Bl -tag -offset indent -width Va
196 IP mode as defined at compile time.
197 An ipmode of "4" means IPv6 is not supported and IP addresses are logged in
198 regular dotted quad format.
199 An ipmode of "6" means IPv6 is supported, and IP addresses are logged in dotted
200 quad or hex format, as described in the
201 .Qq Compile-time Configuration
205 The second type of log message is written to the file when a data log message
207 The text below shows an example data log triggered by an IPv4
209 The data is CSV formatted.
210 .Bd -literal -offset indent
211 o,0xbec491a5,1238556193.463551,172.16.7.28,22,172.16.2.5,55931, \\
212 1073725440,172312,6144,66560,66608,8,1,4,1448,936,1,996,255, \\
213 33304,208,66608,0,208,0
216 Field descriptions are as follows:
217 .Bl -tag -offset indent -width Va
219 Direction of packet that triggered the log message.
226 .Bl -tag -offset indent -width Va
228 Hash of the packet that triggered the log message.
230 .Bl -tag -offset indent -width Va
232 Time at which the packet that triggered the log message was processed by
235 hook function, in seconds and microseconds since the UNIX epoch.
237 .Bl -tag -offset indent -width Va
239 The IPv4 or IPv6 address of the local host, in dotted quad (IPv4 packet)
240 or colon-separated hex (IPv6 packet) notation.
242 .Bl -tag -offset indent -width Va
244 The TCP port that the local host is communicating via.
246 .Bl -tag -offset indent -width Va
248 The IPv4 or IPv6 address of the foreign host, in dotted quad (IPv4 packet)
249 or colon-separated hex (IPv6 packet) notation.
251 .Bl -tag -offset indent -width Va
253 The TCP port that the foreign host is communicating via.
255 .Bl -tag -offset indent -width Va
257 The slow start threshold for the flow, in bytes.
259 .Bl -tag -offset indent -width Va
261 The current congestion window for the flow, in bytes.
263 .Bl -tag -offset indent -width Va
265 The current bandwidth-controlled window for the flow, in bytes.
267 .Bl -tag -offset indent -width Va
269 The current sending window for the flow, in bytes.
270 The post scaled value is reported, except during the initial handshake (first
271 few packets), during which time the unscaled value is reported.
273 .Bl -tag -offset indent -width Va
275 The current receive window for the flow, in bytes.
276 The post scaled value is always reported.
278 .Bl -tag -offset indent -width Va
280 The current window scaling factor for the sending window.
282 .Bl -tag -offset indent -width Va
284 The current window scaling factor for the receiving window.
286 .Bl -tag -offset indent -width Va
288 The current state of the TCP finite state machine, as defined
290 .Aq Pa netinet/tcp_fsm.h .
292 .Bl -tag -offset indent -width Va
294 The maximum segment size for the flow, in bytes.
296 .Bl -tag -offset indent -width Va
298 The current smoothed RTT estimate for the flow, in units of TCP_RTT_SCALE * HZ,
299 where TCP_RTT_SCALE is a define found in tcp_var.h, and HZ is the kernel's tick
301 Divide by TCP_RTT_SCALE * HZ to get the RTT in secs.
302 TCP_RTT_SCALE and HZ are reported in the enable log message.
304 .Bl -tag -offset indent -width Va
306 SACK enabled indicator. 1 if SACK enabled, 0 otherwise.
308 .Bl -tag -offset indent -width Va
310 The current state of the TCP flags for the flow.
312 .Aq Pa netinet/tcp_var.h
313 for information about the various flags.
315 .Bl -tag -offset indent -width Va
317 The current retransmission timeout length for the flow, in units of HZ, where HZ
318 is the kernel's tick timer.
319 Divide by HZ to get the timeout length in seconds.
320 HZ is reported in the enable log message.
322 .Bl -tag -offset indent -width Va
324 The current size of the socket send buffer in bytes.
326 .Bl -tag -offset indent -width Va
328 The current number of bytes in the socket send buffer.
330 .Bl -tag -offset indent -width Va
332 The current size of the socket receive buffer in bytes.
334 .Bl -tag -offset indent -width Va
336 The current number of bytes in the socket receive buffer.
338 .Bl -tag -offset indent -width Va
340 The current number of unacknowledged bytes in-flight.
341 Bytes acknowledged via SACK are not excluded from this count.
343 .Bl -tag -offset indent -width Va
345 The current number of segments in the reassembly queue.
347 .Bl -tag -offset indent -width Va
349 Flowid for the connection.
350 A caveat: Zero '0' either represents a valid flowid or a default value when it's
352 There is no easy way to differentiate without looking at actual
353 network interface card and drivers being used.
355 .Bl -tag -offset indent -width Va
357 Flow type for the connection.
358 Flowtype defines which protocol fields are hashed to produce the flowid.
359 A complete listing is available in
365 The third type of log message is written to the file when the module is disabled
366 and ceases collecting data from the running kernel.
367 The text below shows an example module disable log.
368 The fields are tab delimited key-value pairs which provide statistics about
369 operations since the module was most recently enabled.
370 .Bd -literal -offset indent
371 disable_time_secs=1238556197 disable_time_usecs=933607 \\
372 num_inbound_tcp_pkts=356 num_outbound_tcp_pkts=627 \\
373 total_tcp_pkts=983 num_inbound_skipped_pkts_malloc=0 \\
374 num_outbound_skipped_pkts_malloc=0 num_inbound_skipped_pkts_mtx=0 \\
375 num_outbound_skipped_pkts_mtx=0 num_inbound_skipped_pkts_tcb=0 \\
376 num_outbound_skipped_pkts_tcb=0 num_inbound_skipped_pkts_icb=0 \\
377 num_outbound_skipped_pkts_icb=0 total_skipped_tcp_pkts=0 \\
378 flow_list=172.16.7.28;22-172.16.2.5;55931,
381 Field descriptions are as follows:
382 .Bl -tag -offset indent -width Va
383 .It Va disable_time_secs
384 Time at which the module was disabled, in seconds since the UNIX epoch.
386 .Bl -tag -offset indent -width Va
387 .It Va disable_time_usecs
388 Time at which the module was disabled, in microseconds since disable_time_secs.
390 .Bl -tag -offset indent -width Va
391 .It Va num_inbound_tcp_pkts
392 Number of TCP packets that traversed up the network stack.
393 This only includes inbound TCP packets during the periods when
397 .Bl -tag -offset indent -width Va
398 .It Va num_outbound_tcp_pkts
399 Number of TCP packets that traversed down the network stack.
400 This only includes outbound TCP packets during the periods when
404 .Bl -tag -offset indent -width Va
405 .It Va total_tcp_pkts
406 The summation of num_inbound_tcp_pkts and num_outbound_tcp_pkts.
408 .Bl -tag -offset indent -width Va
409 .It Va num_inbound_skipped_pkts_malloc
410 Number of inbound packets that were not processed because of failed malloc() calls.
412 .Bl -tag -offset indent -width Va
413 .It Va num_outbound_skipped_pkts_malloc
414 Number of outbound packets that were not processed because of failed malloc() calls.
416 .Bl -tag -offset indent -width Va
417 .It Va num_inbound_skipped_pkts_mtx
418 Number of inbound packets that were not processed because of failure to add the
419 packet to the packet processing queue.
421 .Bl -tag -offset indent -width Va
422 .It Va num_outbound_skipped_pkts_mtx
423 Number of outbound packets that were not processed because of failure to add the
424 packet to the packet processing queue.
426 .Bl -tag -offset indent -width Va
427 .It Va num_inbound_skipped_pkts_tcb
428 Number of inbound packets that were not processed because of failure to find the
429 TCP control block associated with the packet.
431 .Bl -tag -offset indent -width Va
432 .It Va num_outbound_skipped_pkts_tcb
433 Number of outbound packets that were not processed because of failure to find
434 the TCP control block associated with the packet.
436 .Bl -tag -offset indent -width Va
437 .It Va num_inbound_skipped_pkts_icb
438 Number of inbound packets that were not processed because of failure to find the
439 IP control block associated with the packet.
441 .Bl -tag -offset indent -width Va
442 .It Va num_outbound_skipped_pkts_icb
443 Number of outbound packets that were not processed because of failure to find
444 the IP control block associated with the packet.
446 .Bl -tag -offset indent -width Va
447 .It Va total_skipped_tcp_pkts
448 The summation of all skipped packet counters.
450 .Bl -tag -offset indent -width Va
452 A CSV list of TCP flows that triggered data log messages to be generated since
453 the module was loaded.
454 Each flow entry in the CSV list is
456 .Qq local_ip;local_port-foreign_ip;foreign_port .
457 If there are no entries in the list (i.e., no data log messages were generated),
458 the value will be blank.
459 If there is at least one entry in the list, a trailing comma will always be
463 The total number of data log messages found in the log file for a module
464 enable/disable cycle should equate to total_tcp_pkts - total_skipped_tcp_pkts.
465 .Sh IMPLEMENTATION NOTES
467 hooks into the network stack using the
470 In its current incarnation, it hooks into the AF_INET/AF_INET6 (IPv4/IPv6)
472 filtering points, which means it sees packets at the IP layer of the network
474 This means that TCP packets inbound to the stack are intercepted before
475 they have been processed by the TCP layer.
476 Packets outbound from the stack are intercepted after they have been processed
479 The diagram below illustrates how
481 inserts itself into the stack.
482 .Bd -literal -offset indent
483 ----------------------------------
485 ----------------------------------
491 ----------------------------------
500 ________| |__________
503 IPv{4/6} in IPv{4/6} out
504 ----------------------------------
508 Layer 2 in Layer 2 out
509 ----------------------------------
511 ----------------------------------
517 interface to manage writing data to disk.
519 At first glance, you might mistakenly think that
521 extracts information from
522 individual TCP packets.
523 This is not the case.
525 uses TCP packet events (inbound and outbound) for each TCP flow originating from
526 the system to trigger a dump of the state of the TCP control block for that
528 With the PPL set to 1, we are in effect sampling each TCP flow's control block
529 state as frequently as flow packets enter/leave the system.
530 For example, setting PPL to 2 halves the sampling rate i.e., every second flow
531 packet (inbound OR outbound) causes a dump of the control block state.
533 The distinction between interrogating individual packets versus interrogating the
534 control block is important, because
536 does not remove the need for packet capturing tools like
539 allows you to correlate and observe the cause-and-affect relationship between
540 what you see on the wire (captured using a tool like
542 and changes in the TCP control block corresponding to the flow of interest.
543 It is therefore useful to use
547 to gather the necessary data to piece together the complete picture.
548 Use of either tool on its own will not be able to provide all of the necessary
551 As a result of needing to interrogate the TCP control block, certain packets
552 during the lifecycle of a connection are unable to trigger a
555 The initial handshake takes place without the existence of a control block and
556 the final ACK is exchanged when the connection is in the TIMEWAIT state.
559 was designed to minimise the delay introduced to packets traversing the network
561 This design called for a highly optimised and minimal hook function that
562 extracted the minimal details necessary whilst holding the packet up, and
563 passing these details to another thread for actual processing and logging.
565 This multithreaded design does introduce some contention issues when accessing
566 the data structure shared between the threads of operation.
567 When the hook function tries to place details in the structure, it must first
568 acquire an exclusive lock.
569 Likewise, when the processing thread tries to read details from the structure,
570 it must also acquire an exclusive lock to do so.
571 If one thread holds the lock, the other must wait before it can obtain it.
572 This does introduce some additional bounded delay into the kernel's packet
573 processing code path.
575 In some cases (e.g., low memory, connection termination), TCP packets that enter
579 hook function will not trigger a log message to be generated.
581 refers to this outcome as a
585 always ensures that packets are allowed to continue through the stack, even if
586 they could not successfully trigger a data log message.
588 will therefore not introduce any packet loss for TCP/IP packets traversing the
590 .Ss Important Behaviours
591 The behaviour of a log file path change whilst the module is enabled is as
595 Attempt to open the new file path for writing.
596 If this fails, the path change will fail and the existing path will continue to
599 Assuming the new path is valid and opened successfully:
602 Flush all pending log messages to the old file path.
604 Close the old file path.
606 Switch the active log file pointer to point at the new file path.
608 Commence logging to the new file.
612 During the time between the flush of pending log messages to the old file and
613 commencing logging to the new file, new log messages will still be generated and
615 As soon as the new file path is ready for writing, the accumulated log messages
616 will be written out to the file.
618 To enable the module's operations, run the following command as root:
619 sysctl net.inet.siftr.enabled=1
621 To change the granularity of log messages such that 1 log message is
622 generated for every 10 TCP packets per connection, run the following
624 sysctl net.inet.siftr.ppl=10
626 To change the log file location to /tmp/siftr.log, run the following
628 sysctl net.inet.siftr.logfile=/tmp/siftr.log
636 Development of this software was made possible in part by grants from the
637 Cisco University Research Program Fund at Community Foundation Silicon Valley,
638 and the FreeBSD Foundation.
647 was first released in 2007 by Lawrence Stewart and James Healy whilst working on
648 the NewTCP research project at Swinburne University of Technology's Centre for
649 Advanced Internet Architectures, Melbourne, Australia, which was made possible
650 in part by a grant from the Cisco University Research Program Fund at Community
651 Foundation Silicon Valley.
652 More details are available at:
654 http://caia.swin.edu.au/urp/newtcp/
658 v1.2.x was sponsored by the FreeBSD Foundation as part of
660 .Qq Enhancing the FreeBSD TCP Implementation
662 More details are available at:
664 http://www.freebsdfoundation.org/
666 http://caia.swin.edu.au/freebsd/etcp09/
671 .An Lawrence Stewart Aq Mt lstewart@FreeBSD.org
673 .An James Healy Aq Mt jimmy@deefa.com .
675 This manual page was written by
676 .An Lawrence Stewart Aq Mt lstewart@FreeBSD.org .
678 Current known limitations and any relevant workarounds are outlined below:
681 The internal queue used to pass information between the threads of operation is
685 to cope with bursty network traffic, but sustained high packet-per-second
686 traffic can cause exhaustion of kernel memory if the processing thread cannot
687 keep up with the packet rate.
691 on a machine that is also running other modules utilising the
697 the order in which you load the modules is important.
698 You should kldload the other modules first, as this will ensure TCP packets
699 undergo any necessary manipulations before
704 There is a known, harmless lock order reversal warning between the
706 mutex and tcbinfo TCP lock reported by
710 is enabled in a kernel compiled with
714 There is no way to filter which TCP flows you wish to capture data for.
715 Post processing is required to separate out data belonging to particular flows
718 The module does not detect deletion of the log file path.
719 New log messages will simply be lost if the log file being used by
721 is deleted whilst the module is set to use the file.
722 Switching to a new log file using the
723 .Em net.inet.siftr.logfile
724 variable will create the new file and allow log messages to begin being written
726 The new log file path must differ from the path to the deleted file.
728 The hash table used within the code is sized to hold 65536 flows.
730 hard limit, because chaining is used to handle collisions within the hash table
732 However, we suspect (based on analogies with other hash table performance data)
733 that the hash table look up performance (and therefore the module's packet
734 processing performance) will degrade in an exponential manner as the number of
735 unique flows handled in a module enable/disable cycle approaches and surpasses
738 There is no garbage collection performed on the flow hash table.
739 The only way currently to flush it is to disable
742 The PPL variable applies to packets that make it into the processing thread,
743 not total packets received in the hook function.
744 Packets are skipped before the PPL variable is applied, which means there may be
745 a slight discrepancy in the triggering of log messages.
746 For example, if PPL was set to 10, and the 8th packet since the last log message
747 is skipped, the 11th packet will actually trigger the log message to be
749 This is discussed in greater depth in CAIA technical report 070824A.
751 At the time of writing, there was no simple way to hook into the TCP layer
752 to intercept packets.
754 use of IP layer hook points means all IP
755 traffic will be processed by the
758 hook function, which introduces minor, but nonetheless unnecessary packet delay
759 and processing overhead on the system for non-TCP packets as well.
760 Hooking in at the IP layer is also not ideal from the data gathering point of
762 Packets traversing up the stack will be intercepted and cause a log message
763 generation BEFORE they have been processed by the TCP layer, which means we
764 cannot observe the cause-and-affect relationship between inbound events and the
765 corresponding TCP control block as precisely as could be.
768 should intercept packets after they have been processed by the TCP layer i.e.
769 intercept packets coming up the stack after they have been processed by
770 tcp_input(), and intercept packets coming down the stack after they have been
771 processed by tcp_output().
772 The current code still gives satisfactory granularity though, as inbound events
773 tend to trigger outbound events, allowing the cause-and-effect to be observed
774 indirectly by capturing the state on outbound events as well.
780 does not take into account bytes that have been
782 by the receiving host.
784 Packet hash generation does not currently work for IPv6 based TCP packets.
786 Compressed notation is not used for IPv6 address representation.
787 This consumes more bytes than is necessary in log output.