1 .\" Copyright (c) 2011-2014 Matteo Landi, Luigi Rizzo, Universita` di Pisa
2 .\" All rights reserved.
4 .\" Redistribution and use in source and binary forms, with or without
5 .\" modification, are permitted provided that the following conditions
7 .\" 1. Redistributions of source code must retain the above copyright
8 .\" notice, this list of conditions and the following disclaimer.
9 .\" 2. Redistributions in binary form must reproduce the above copyright
10 .\" notice, this list of conditions and the following disclaimer in the
11 .\" documentation and/or other materials provided with the distribution.
13 .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
14 .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
15 .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
16 .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
17 .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
18 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
19 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
20 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
21 .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
22 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
25 .\" This document is derived in part from the enet man page (enet.4)
26 .\" distributed with 4.3BSD Unix.
35 .Nd a framework for fast packet I/O
37 .Nd a fast VirtuAl Local Ethernet using the netmap API
40 .Nd a shared memory packet transport channel
45 is a framework for extremely fast and efficient packet I/O
46 for userspace and kernel clients, and for Virtual Machines.
49 Linux and some versions of Windows, and supports a variety of
53 .It Nm physical NIC ports
54 to access individual queues of network interfaces;
56 to inject packets into the host stack;
58 implementing a very fast and modular in-kernel software switch/dataplane;
60 a shared memory packet transport channel;
61 .It Nm netmap monitors
62 a mechanism similar to
69 are accessed interchangeably with the same API,
70 and are at least one order of magnitude faster than
71 standard OS mechanisms
72 (sockets, bpf, tun/tap interfaces, native switches, pipes).
73 With suitably fast hardware (NICs, PCIe buses, CPUs),
77 reaches 14.88 million packets per second (Mpps)
78 with much less than one core on 10 Gbit/s NICs;
79 35-40 Mpps on 40 Gbit/s NICs (limited by the hardware);
80 about 20 Mpps per core for VALE ports;
85 support can still use the API in emulated mode,
86 which uses unmodified device drivers and is 3-5 times faster than
90 Userspace clients can dynamically switch NICs into
92 mode and send and receive raw packets through
93 memory mapped buffers.
96 switch instances and ports,
100 can be created dynamically,
101 providing high speed packet I/O between processes,
102 virtual machines, NICs and the host stack.
105 supports both non-blocking I/O through
107 synchronization and blocking I/O through a file descriptor
108 and standard OS mechanisms such as
118 are implemented by a single kernel module, which also emulates the
120 API over standard drivers.
121 For best performance,
123 requires native support in device drivers.
124 A list of such devices is at the end of this document.
126 In the rest of this (long) manual page we document
127 various aspects of the
131 architecture, features and usage.
134 supports raw packet I/O through a
136 which can be connected to a physical interface
142 Ports use preallocated circular queues of buffers
144 residing in an mmapped region.
145 There is one ring for each transmit/receive queue of a
147 An additional ring pair connects to the host stack.
149 After binding a file descriptor to a port, a
151 client can send or receive packets in batches through
152 the rings, and possibly implement zero-copy forwarding
155 All NICs operating in
157 mode use the same memory region,
158 accessible to all processes who own
160 file descriptors bound to NICs.
166 by default use separate memory regions,
167 but can be independently configured to share memory.
168 .Sh ENTERING AND EXITING NETMAP MODE
169 The following section describes the system calls to create
177 Simpler, higher level functions are described in the
181 Ports and rings are created and controlled through a file descriptor,
182 created by opening a special device
183 .Dl fd = open("/dev/netmap");
184 and then bound to a specific port with an
185 .Dl ioctl(fd, NIOCREGIF, (struct nmreq *)arg);
188 has multiple modes of operation controlled by the
192 specifies the netmap port name, as follows:
194 .It Dv OS network interface name (e.g., 'em0', 'eth1', ... )
195 the data path of the NIC is disconnected from the host stack,
196 and the file descriptor is bound to the NIC (one or all queues),
197 or to the host stack;
199 the file descriptor is bound to port PPP of VALE switch SSS.
200 Switch instances and ports are dynamically created if necessary.
202 Both SSS and PPP have the form [0-9a-zA-Z_]+ , the string
203 cannot exceed IFNAMSIZ characters, and PPP cannot
204 be the name of any existing OS network interface.
209 indicates the size of the shared memory region,
210 and the number, size and location of all the
212 data structures, which can be accessed by mmapping the memory
213 .Dl char *mem = mmap(0, arg.nr_memsize, fd);
215 Non-blocking I/O is done with special
220 on the file descriptor permit blocking I/O.
230 mode, the OS will still believe the interface is up and running.
231 OS-generated packets for that NIC end up into a
233 ring, and another ring is used to send packets into the OS network stack.
236 on the file descriptor removes the binding,
237 and returns the NIC to normal mode (reconnecting the data path
238 to the host stack), or destroys the virtual port.
240 The data structures in the mmapped memory region are detailed in
241 .In sys/net/netmap.h ,
242 which is the ultimate reference for the
245 The main structures and fields are indicated below:
247 .It Dv struct netmap_if (one per interface)
251 const uint32_t ni_flags; /* properties */
253 const uint32_t ni_tx_rings; /* NIC tx rings */
254 const uint32_t ni_rx_rings; /* NIC rx rings */
255 uint32_t ni_bufs_head; /* head of extra bufs list */
260 Indicates the number of available rings
261 .Pa ( struct netmap_rings )
262 and their position in the mmapped region.
263 The number of tx and rx rings
264 .Pa ( ni_tx_rings , ni_rx_rings )
265 normally depends on the hardware.
266 NICs also have an extra tx/rx ring pair connected to the host stack.
268 can also request additional unbound buffers in the same memory space,
269 to be used as temporary storage for packets.
271 contains the index of the first of these free rings,
272 which are connected in a list (the first uint32_t of each
273 buffer being the index of the next buffer in the list).
276 indicates the end of the list.
277 .It Dv struct netmap_ring (one per ring)
281 const uint32_t num_slots; /* slots in each ring */
282 const uint32_t nr_buf_size; /* size of each buffer */
284 uint32_t head; /* (u) first buf owned by user */
285 uint32_t cur; /* (u) wakeup position */
286 const uint32_t tail; /* (k) first buf owned by kernel */
289 struct timeval ts; /* (k) time of last rxsync() */
291 struct netmap_slot slot[0]; /* array of slots */
295 Implements transmit and receive rings, with read/write
296 pointers, metadata and an array of
298 describing the buffers.
299 .It Dv struct netmap_slot (one per buffer)
302 uint32_t buf_idx; /* buffer index */
303 uint16_t len; /* packet length */
304 uint16_t flags; /* buf changed, etc. */
305 uint64_t ptr; /* address for indirect buffers */
309 Describes a packet buffer, which normally is identified by
310 an index and resides in the mmapped region.
311 .It Dv packet buffers
312 Fixed size (normally 2 KB) packet buffers allocated by the kernel.
317 in the mmapped region is indicated by the
319 field in the structure returned by
321 From there, all other objects are reachable through
322 relative references (offsets or indexes).
323 Macros and functions in
324 .In net/netmap_user.h
325 help converting them into actual pointers:
327 .Dl struct netmap_if *nifp = NETMAP_IF(mem, arg.nr_offset);
328 .Dl struct netmap_ring *txr = NETMAP_TXRING(nifp, ring_index);
329 .Dl struct netmap_ring *rxr = NETMAP_RXRING(nifp, ring_index);
331 .Dl char *buf = NETMAP_BUF(ring, buffer_index);
332 .Sh RINGS, BUFFERS AND DATA I/O
334 are circular queues of packets with three indexes/pointers
335 .Va ( head , cur , tail ) ;
336 one slot is always kept empty.
339 should not be assumed to be a power of two.
342 is the first slot available to userspace;
346 select/poll will unblock when
352 is the first slot reserved to the kernel.
357 for convenience, the function
358 .Dl nm_ring_next(ring, index)
359 returns the next index modulo the ring size.
364 are only modified by the user program;
366 is only modified by the kernel.
367 The kernel only reads/writes the
368 .Vt struct netmap_ring
370 during the execution of a netmap-related system call.
371 The only exception are slots (and buffers) in the range
372 .Va tail\ . . . head-1 ,
373 that are explicitly assigned to the kernel.
376 On transmit rings, after a
378 system call, slots in the range
379 .Va head\ . . . tail-1
380 are available for transmission.
381 User code should fill the slots sequentially
386 past slots ready to transmit.
388 may be moved further ahead if the user code needs
389 more slots before further transmissions (see
390 .Sx SCATTER GATHER I/O ) .
392 At the next NIOCTXSYNC/select()/poll(),
395 are pushed to the port, and
397 may advance if further slots have become available.
398 Below is an example of the evolution of a TX ring:
400 after the syscall, slots between cur and tail are (a)vailable
404 TX [.....aaaaaaaaaaa.............]
406 user creates new packets to (T)ransmit
410 TX [.....TTTTTaaaaaa.............]
412 NIOCTXSYNC/poll()/select() sends packets and reports new slots
416 TX [..........aaaaaaaaaaa........]
422 will block if there is no space in the ring, i.e.,
423 .Dl ring->cur == ring->tail
424 and return when new slots have become available.
426 High speed applications may want to amortize the cost of system calls
427 by preparing as many packets as possible before issuing them.
429 A transmit ring with pending transmissions has
430 .Dl ring->head != ring->tail + 1 (modulo the ring size).
432 .Va int nm_tx_pending(ring)
433 implements this test.
435 On receive rings, after a
437 system call, the slots in the range
438 .Va head\& . . . tail-1
439 contain received packets.
440 User code should process them and advance
444 past slots it wants to return to the kernel.
446 may be moved further ahead if the user code wants to
447 wait for more packets
448 without returning all the previous slots to the kernel.
450 At the next NIOCRXSYNC/select()/poll(),
453 are returned to the kernel for further receives, and
455 may advance to report new incoming packets.
457 Below is an example of the evolution of an RX ring:
459 after the syscall, there are some (h)eld and some (R)eceived slots
463 RX [..hhhhhhRRRRRRRR..........]
465 user advances head and cur, releasing some slots and holding others
469 RX [..*****hhhRRRRRR...........]
471 NICRXSYNC/poll()/select() recovers slots and reports new packets
475 RX [.......hhhRRRRRRRRRRRR....]
477 .Sh SLOTS AND PACKET BUFFERS
478 Normally, packets should be stored in the netmap-allocated buffers
479 assigned to slots when ports are bound to a file descriptor.
480 One packet is fully contained in a single buffer.
482 The following flags affect slot and buffer processing:
488 in the slot is changed.
489 This can be used to implement
490 zero-copy forwarding, see
491 .Sx ZERO-COPY FORWARDING .
493 reports when this buffer has been transmitted.
496 notifies transmit completions in batches, hence signals
497 can be delayed indefinitely.
498 This flag helps detect
499 when packets have been sent and a file descriptor can be closed.
501 When a ring is in 'transparent' mode (see
502 .Sx TRANSPARENT MODE ) ,
503 packets marked with this flag are forwarded to the other endpoint
504 at the next system call, thus restoring (in a selective way)
505 the connection between a NIC and the host stack.
507 tells the forwarding code that the source MAC address for this
508 packet must not be used in the learning bridge code.
510 indicates that the packet's payload is in a user-supplied buffer
511 whose user virtual address is in the 'ptr' field of the slot.
512 The size can reach 65535 bytes.
514 This is only supported on the transmit ring of
516 ports, and it helps reducing data copies in the interconnection
519 indicates that the packet continues with subsequent buffers;
520 the last buffer in a packet must have the flag clear.
522 .Sh SCATTER GATHER I/O
523 Packets can span multiple slots if the
525 flag is set in all but the last slot.
526 The maximum length of a chain is 64 buffers.
527 This is normally used with
529 ports when connecting virtual machines, as they generate large
530 TSO segments that are not split unless they reach a physical device.
532 NOTE: The length field always refers to the individual
533 fragment; there is no place with the total length of a packet.
535 On receive rings the macro
537 indicates the remaining number of slots for this packet,
538 including the current one.
539 Slots with a value greater than 1 also have NS_MOREFRAG set.
542 uses two ioctls (NIOCTXSYNC, NIOCRXSYNC)
543 for non-blocking I/O.
544 They take no argument.
545 Two more ioctls (NIOCGINFO, NIOCREGIF) are used
546 to query and configure ports, with the following argument:
549 char nr_name[IFNAMSIZ]; /* (i) port name */
550 uint32_t nr_version; /* (i) API version */
551 uint32_t nr_offset; /* (o) nifp offset in mmap region */
552 uint32_t nr_memsize; /* (o) size of the mmap region */
553 uint32_t nr_tx_slots; /* (i/o) slots in tx rings */
554 uint32_t nr_rx_slots; /* (i/o) slots in rx rings */
555 uint16_t nr_tx_rings; /* (i/o) number of tx rings */
556 uint16_t nr_rx_rings; /* (i/o) number of rx rings */
557 uint16_t nr_ringid; /* (i/o) ring(s) we care about */
558 uint16_t nr_cmd; /* (i) special command */
559 uint16_t nr_arg1; /* (i/o) extra arguments */
560 uint16_t nr_arg2; /* (i/o) extra arguments */
561 uint32_t nr_arg3; /* (i/o) extra arguments */
562 uint32_t nr_flags /* (i/o) open mode */
567 A file descriptor obtained through
569 also supports the ioctl supported by network devices, see
573 returns EINVAL if the named port does not support netmap.
574 Otherwise, it returns 0 and (advisory) information
576 Note that all the information below can change before the
577 interface is actually put in netmap mode.
580 indicates the size of the
585 mode all share the same memory region,
588 ports have independent regions for each port.
589 .It Pa nr_tx_slots , nr_rx_slots
590 indicate the size of transmit and receive rings.
591 .It Pa nr_tx_rings , nr_rx_rings
592 indicate the number of transmit
594 Both ring number and sizes may be configured at runtime
595 using interface-specific functions (e.g.,
600 binds the port named in
602 to the file descriptor.
603 For a physical device this also switches it into
606 it from the host stack.
607 Multiple file descriptors can be bound to the same port,
608 with proper synchronization left to the user.
610 The recommended way to bind a file descriptor to a port is
615 which parses names to access specific port types and
617 In the following we document the main features.
619 .Dv NIOCREGIF can also bind a file descriptor to one endpoint of a
621 consisting of two netmap ports with a crossover connection.
622 A netmap pipe share the same memory space of the parent port,
623 and is meant to enable configuration where a master process acts
624 as a dispatcher towards slave processes.
626 To enable this function, the
628 field of the structure can be used as a hint to the kernel to
629 indicate how many pipes we expect to use, and reserve extra space
630 in the memory region.
632 On return, it gives the same info as NIOCGINFO,
637 indicating the identity of the rings controlled through the file
642 selects which rings are controlled through this file descriptor.
645 are indicated below, together with the naming schemes
646 that application libraries (such as the
648 indicated below) can use to indicate the specific set of rings.
649 In the example below, "netmap:foo" is any valid netmap port name.
650 .Bl -tag -width XXXXX
651 .It NR_REG_ALL_NIC "netmap:foo"
652 (default) all hardware ring pairs
653 .It NR_REG_SW "netmap:foo^"
654 the ``host rings'', connecting to the host stack.
655 .It NR_REG_NIC_SW "netmap:foo+"
656 all hardware rings and the host rings
657 .It NR_REG_ONE_NIC "netmap:foo-i"
658 only the i-th hardware ring pair, where the number is in
660 .It NR_REG_PIPE_MASTER "netmap:foo{i"
661 the master side of the netmap pipe whose identifier (i) is in
663 .It NR_REG_PIPE_SLAVE "netmap:foo}i"
664 the slave side of the netmap pipe whose identifier (i) is in
667 The identifier of a pipe must be thought as part of the pipe name,
668 and does not need to be sequential.
670 will only have a single ring pair with index 0,
671 irrespective of the value of
679 call pushes out any pending packets on the transmit ring, even if
680 no write events are specified.
681 The feature can be disabled by or-ing
682 .Va NETMAP_NO_TX_POLL
683 to the value written to
685 When this feature is used,
686 packets are transmitted only on
687 .Va ioctl(NIOCTXSYNC)
688 or select()/poll() are called with a write event (POLLOUT/wfdset) or a full ring.
690 When registering a virtual interface that is dynamically created to a
692 switch, we can specify the desired number of rings (1 by default,
693 and currently up to 16) on it using nr_tx_rings and nr_rx_rings fields.
695 tells the hardware of new packets to transmit, and updates the
696 number of slots available for transmission.
698 tells the hardware of consumed packets, and asks for newly available
701 .Sh SELECT, POLL, EPOLL, KQUEUE.
707 file descriptor process rings as indicated in
711 respectively when write (POLLOUT) and read (POLLIN) events are requested.
712 Both block if no slots are available in the ring
713 .Va ( ring->cur == ring->tail ) .
714 Depending on the platform,
720 Packets in transmit rings are normally pushed out
721 (and buffers reclaimed) even without
722 requesting write events.
724 .Dv NETMAP_NO_TX_POLL
727 disables this feature.
728 By default, receive rings are processed only if read
729 events are requested.
731 .Dv NETMAP_DO_RX_POLL
733 .Em NIOCREGIF updates receive rings even without read events.
734 Note that on epoll and kqueue,
735 .Dv NETMAP_NO_TX_POLL
737 .Dv NETMAP_DO_RX_POLL
738 only have an effect when some event is posted for the file descriptor.
742 API is supposed to be used directly, both because of its simplicity and
743 for efficient integration with applications.
746 .In net/netmap_user.h
747 header provides a few macros and functions to ease creating
748 a file descriptor and doing I/O with a
751 These are loosely modeled after the
753 API, to ease porting of libpcap-based applications to
755 To use these extra functions, programs should
756 .Dl #define NETMAP_WITH_LIBS
758 .Dl #include <net/netmap_user.h>
760 The following functions are available:
761 .Bl -tag -width XXXXX
762 .It Va struct nm_desc * nm_open(const char *ifname, const struct nmreq *req, uint64_t flags, const struct nm_desc *arg)
764 .Xr pcap_open 3pcap ,
765 binds a file descriptor to a port.
768 is a port name, in the form "netmap:PPP" for a NIC and "valeSSS:PPP" for a
772 provides the initial values for the argument to the NIOCREGIF ioctl.
773 The nm_flags and nm_ringid values are overwritten by parsing
774 ifname and flags, and other fields can be overridden through
775 the other two arguments.
777 points to a struct nm_desc containing arguments (e.g., from a previously
778 open file descriptor) that should override the defaults.
779 The fields are used as described below
781 can be set to a combination of the following flags:
782 .Va NETMAP_NO_TX_POLL ,
783 .Va NETMAP_DO_RX_POLL
784 (copied into nr_ringid);
785 .Va NM_OPEN_NO_MMAP (if arg points to the same memory region,
786 avoids the mmap and uses the values from it);
787 .Va NM_OPEN_IFNAME (ignores ifname and uses the values in arg);
790 .Va NM_OPEN_ARG3 (uses the fields from arg);
791 .Va NM_OPEN_RING_CFG (uses the ring number and sizes from arg).
793 .It Va int nm_close(struct nm_desc *d)
794 closes the file descriptor, unmaps memory, frees resources.
795 .It Va int nm_inject(struct nm_desc *d, const void *buf, size_t size)
796 similar to pcap_inject(), pushes a packet to a ring, returns the size
797 of the packet is successful, or 0 on error;
798 .It Va int nm_dispatch(struct nm_desc *d, int cnt, nm_cb_t cb, u_char *arg)
799 similar to pcap_dispatch(), applies a callback to incoming packets
800 .It Va u_char * nm_nextpkt(struct nm_desc *d, struct nm_pkthdr *hdr)
801 similar to pcap_next(), fetches the next packet
803 .Sh SUPPORTED DEVICES
805 natively supports the following devices:
824 NICs without native support can still be used in
826 mode through emulation.
827 Performance is inferior to native netmap
828 mode but still significantly higher than various raw socket types
829 (bpf, PF_PACKET, etc.).
830 Note that for slow devices (such as 1 Gbit/s and slower NICs,
831 or several 10 Gbit/s NICs whose hardware is unable to sustain line rate),
832 emulated and native mode will likely have similar or same throughput.
834 When emulation is in use, packet sniffer programs such as tcpdump
835 could see received packets before they are diverted by netmap.
836 This behaviour is not intentional, being just an artifact of the implementation
838 Note that in case the netmap application subsequently moves packets received
839 from the emulated adapter onto the host RX ring, the sniffer will intercept
840 those packets again, since the packets are injected to the host stack as they
841 were received by the network interface.
843 Emulation is also available for devices with native netmap support,
844 which can be used for testing or performance comparison.
846 .Va dev.netmap.admode
847 globally controls how netmap mode is implemented.
848 .Sh SYSCTL VARIABLES AND MODULE PARAMETERS
849 Some aspect of the operation of
851 are controlled through sysctl variables on FreeBSD
853 and module parameters on Linux
854 .Em ( /sys/module/netmap_lin/parameters/* ) :
855 .Bl -tag -width indent
856 .It Va dev.netmap.admode: 0
857 Controls the use of native or emulated adapter mode.
859 0 uses the best available option;
861 1 forces native mode and fails if not available;
863 2 forces emulated hence never fails.
864 .It Va dev.netmap.generic_ringsize: 1024
865 Ring size used for emulated netmap mode
866 .It Va dev.netmap.generic_mit: 100000
867 Controls interrupt moderation for emulated mode
868 .It Va dev.netmap.mmap_unreg: 0
869 .It Va dev.netmap.fwd: 0
870 Forces NS_FORWARD mode
871 .It Va dev.netmap.flags: 0
872 .It Va dev.netmap.txsync_retry: 2
873 .It Va dev.netmap.no_pendintr: 1
874 Forces recovery of transmit buffers on system calls
875 .It Va dev.netmap.mitigate: 1
876 Propagates interrupt mitigation to user processes
877 .It Va dev.netmap.no_timestamp: 0
878 Disables the update of the timestamp in the netmap ring
879 .It Va dev.netmap.verbose: 0
880 Verbose kernel messages
881 .It Va dev.netmap.buf_num: 163840
882 .It Va dev.netmap.buf_size: 2048
883 .It Va dev.netmap.ring_num: 200
884 .It Va dev.netmap.ring_size: 36864
885 .It Va dev.netmap.if_num: 100
886 .It Va dev.netmap.if_size: 1024
887 Sizes and number of objects (netmap_if, netmap_ring, buffers)
888 for the global memory region.
889 The only parameter worth modifying is
890 .Va dev.netmap.buf_num
891 as it impacts the total amount of memory used by netmap.
892 .It Va dev.netmap.buf_curr_num: 0
893 .It Va dev.netmap.buf_curr_size: 0
894 .It Va dev.netmap.ring_curr_num: 0
895 .It Va dev.netmap.ring_curr_size: 0
896 .It Va dev.netmap.if_curr_num: 0
897 .It Va dev.netmap.if_curr_size: 0
898 Actual values in use.
899 .It Va dev.netmap.bridge_batch: 1024
900 Batch size used when moving packets across a
903 Values above 64 generally guarantee good
914 to wake up processes when significant events occur, and
918 is used to configure ports and
921 Applications may need to create threads and bind them to
922 specific cores to improve performance, using standard
926 .Xr pthread_setaffinity_np 3
931 comes with a few programs that can be used for testing or
938 .Pa tools/tools/netmap/
944 is a general purpose traffic source/sink.
947 .Dl pkt-gen -i ix0 -f tx -l 60
948 can generate an infinite stream of minimum size packets, and
949 .Dl pkt-gen -i ix0 -f rx
951 Both print traffic statistics, to help monitor
952 how the system performs.
955 has many options can be uses to set packet sizes, addresses,
956 rates, and use multiple send/receive threads and cores.
959 is another test program which interconnects two
962 It can be used for transparent forwarding between
964 .Dl bridge -i ix0 -i ix1
965 or even connect the NIC to the host stack using netmap
966 .Dl bridge -i ix0 -i ix0
967 .Ss USING THE NATIVE API
968 The following code implements a traffic generator
970 .Bd -literal -compact
971 #include <net/netmap_user.h>
975 struct netmap_if *nifp;
976 struct netmap_ring *ring;
980 fd = open("/dev/netmap", O_RDWR);
981 bzero(&nmr, sizeof(nmr));
982 strcpy(nmr.nr_name, "ix0");
983 nmr.nm_version = NETMAP_API;
984 ioctl(fd, NIOCREGIF, &nmr);
985 p = mmap(0, nmr.nr_memsize, fd);
986 nifp = NETMAP_IF(p, nmr.nr_offset);
987 ring = NETMAP_TXRING(nifp, 0);
989 fds.events = POLLOUT;
992 while (!nm_ring_empty(ring)) {
994 buf = NETMAP_BUF(ring, ring->slot[i].buf_index);
995 ... prepare packet in buf ...
996 ring->slot[i].len = ... packet length ...
997 ring->head = ring->cur = nm_ring_next(ring, i);
1002 .Ss HELPER FUNCTIONS
1003 A simple receiver can be implemented using the helper functions
1004 .Bd -literal -compact
1005 #define NETMAP_WITH_LIBS
1006 #include <net/netmap_user.h>
1015 d = nm_open("netmap:ix0", NULL, 0, 0);
1016 fds.fd = NETMAP_FD(d);
1017 fds.events = POLLIN;
1020 while ( (buf = nm_nextpkt(d, &h)) )
1021 consume_pkt(buf, h->len);
1026 .Ss ZERO-COPY FORWARDING
1027 Since physical interfaces share the same memory region,
1028 it is possible to do packet forwarding between ports
1030 The buffer from the transmit ring is used
1031 to replenish the receive ring:
1032 .Bd -literal -compact
1034 struct netmap_slot *src, *dst;
1036 src = &src_ring->slot[rxr->cur];
1037 dst = &dst_ring->slot[txr->cur];
1039 dst->buf_idx = src->buf_idx;
1040 dst->len = src->len;
1041 dst->flags = NS_BUF_CHANGED;
1043 src->flags = NS_BUF_CHANGED;
1044 rxr->head = rxr->cur = nm_ring_next(rxr, rxr->cur);
1045 txr->head = txr->cur = nm_ring_next(txr, txr->cur);
1048 .Ss ACCESSING THE HOST STACK
1049 The host stack is for all practical purposes just a regular ring pair,
1050 which you can access with the netmap API (e.g., with
1051 .Dl nm_open("netmap:eth0^", ... ) ;
1052 All packets that the host would send to an interface in
1054 mode end up into the RX ring, whereas all packets queued to the
1055 TX ring are send up to the host stack.
1057 A simple way to test the performance of a
1059 switch is to attach a sender and a receiver to it,
1060 e.g., running the following in two different terminals:
1061 .Dl pkt-gen -i vale1:a -f rx # receiver
1062 .Dl pkt-gen -i vale1:b -f tx # sender
1063 The same example can be used to test netmap pipes, by simply
1064 changing port names, e.g.,
1065 .Dl pkt-gen -i vale2:x{3 -f rx # receiver on the master side
1066 .Dl pkt-gen -i vale2:x}3 -f tx # sender on the slave side
1068 The following command attaches an interface and the host stack
1070 .Dl vale-ctl -h vale2:em0
1073 clients attached to the same switch can now communicate
1074 with the network card or the host.
1082 .Pa http://info.iet.unipi.it/~luigi/netmap/
1084 Luigi Rizzo, Revisiting network I/O APIs: the netmap framework,
1085 Communications of the ACM, 55 (3), pp.45-51, March 2012
1087 Luigi Rizzo, netmap: a novel framework for fast packet I/O,
1088 Usenix ATC'12, June 2012, Boston
1090 Luigi Rizzo, Giuseppe Lettieri,
1091 VALE, a switched ethernet for virtual machines,
1092 ACM CoNEXT'12, December 2012, Nice
1094 Luigi Rizzo, Giuseppe Lettieri, Vincenzo Maffione,
1095 Speeding up packet I/O in virtual machines,
1096 ACM/IEEE ANCS'13, October 2013, San Jose
1101 framework has been originally designed and implemented at the
1102 Universita` di Pisa in 2011 by
1104 and further extended with help from
1106 .An Gaetano Catalli ,
1107 .An Giuseppe Lettieri ,
1109 .An Vincenzo Maffione .
1114 have been funded by the European Commission within FP7 Projects
1115 CHANGE (257422) and OPENLAB (287581).
1117 No matter how fast the CPU and OS are,
1118 achieving line rate on 10G and faster interfaces
1119 requires hardware with sufficient performance.
1120 Several NICs are unable to sustain line rate with
1122 Insufficient PCIe or memory bandwidth
1123 can also cause reduced performance.
1125 Another frequent reason for low performance is the use
1126 of flow control on the link: a slow receiver can limit
1128 Be sure to disable flow control when running high
1130 .Ss SPECIAL NIC FEATURES
1132 is orthogonal to some NIC features such as
1133 multiqueue, schedulers, packet filters.
1135 Multiple transmit and receive rings are supported natively
1136 and can be configured with ordinary OS tools,
1140 device-specific sysctl variables.
1141 The same goes for Receive Packet Steering (RPS)
1142 and filtering of incoming traffic.
1147 .Em checksum offloading , TCP segmentation offloading ,
1148 .Em encryption , VLAN encapsulation/decapsulation ,
1150 When using netmap to exchange packets with the host stack,
1151 make sure to disable these features.