1 .\" Copyright (c) 2011-2014 Matteo Landi, Luigi Rizzo, Universita` di Pisa
2 .\" All rights reserved.
4 .\" Redistribution and use in source and binary forms, with or without
5 .\" modification, are permitted provided that the following conditions
7 .\" 1. Redistributions of source code must retain the above copyright
8 .\" notice, this list of conditions and the following disclaimer.
9 .\" 2. Redistributions in binary form must reproduce the above copyright
10 .\" notice, this list of conditions and the following disclaimer in the
11 .\" documentation and/or other materials provided with the distribution.
13 .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
14 .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
15 .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
16 .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
17 .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
18 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
19 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
20 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
21 .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
22 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
25 .\" This document is derived in part from the enet man page (enet.4)
26 .\" distributed with 4.3BSD Unix.
35 .Nd a framework for fast packet I/O
38 .Nd a fast VirtuAl Local Ethernet using the netmap API
41 .Nd a shared memory packet transport channel
46 is a framework for extremely fast and efficient packet I/O
47 for both userspace and kernel clients.
48 It runs on FreeBSD and Linux,
51 a very fast and modular in-kernel software switch/dataplane,
54 a shared memory packet transport channel.
55 All these are accessed interchangeably with the same API.
60 are at least one order of magnitude faster than
61 standard OS mechanisms
62 (sockets, bpf, tun/tap interfaces, native switches, pipes),
63 reaching 14.88 million packets per second (Mpps)
64 with much less than one core on a 10 Gbit NIC,
65 about 20 Mpps per core for VALE ports,
66 and over 100 Mpps for netmap pipes.
68 Userspace clients can dynamically switch NICs into
70 mode and send and receive raw packets through
71 memory mapped buffers.
74 switch instances and ports, and
76 can be created dynamically,
77 providing high speed packet I/O between processes,
78 virtual machines, NICs and the host stack.
81 suports both non-blocking I/O through
83 synchronization and blocking I/O through a file descriptor
84 and standard OS mechanisms such as
92 are implemented by a single kernel module, which also emulates the
94 API over standard drivers for devices without native
99 requires explicit support in device drivers.
101 In the rest of this (long) manual page we document
102 various aspects of the
106 architecture, features and usage.
109 supports raw packet I/O through a
111 which can be connected to a physical interface
117 Ports use preallocated circular queues of buffers
119 residing in an mmapped region.
120 There is one ring for each transmit/receive queue of a
122 An additional ring pair connects to the host stack.
124 After binding a file descriptor to a port, a
126 client can send or receive packets in batches through
127 the rings, and possibly implement zero-copy forwarding
130 All NICs operating in
132 mode use the same memory region,
133 accessible to all processes who own
135 file descriptors bound to NICs.
141 by default use separate memory regions,
142 but can be independently configured to share memory.
143 .Sh ENTERING AND EXITING NETMAP MODE
144 The following section describes the system calls to create
152 Simpler, higher level functions are described in section
155 Ports and rings are created and controlled through a file descriptor,
156 created by opening a special device
157 .Dl fd = open("/dev/netmap");
158 and then bound to a specific port with an
159 .Dl ioctl(fd, NIOCREGIF, (struct nmreq *)arg);
162 has multiple modes of operation controlled by the
166 specifies the port name, as follows:
168 .It Dv OS network interface name (e.g. 'em0', 'eth1', ... )
169 the data path of the NIC is disconnected from the host stack,
170 and the file descriptor is bound to the NIC (one or all queues),
171 or to the host stack;
172 .It Dv valeXXX:YYY (arbitrary XXX and YYY)
173 the file descriptor is bound to port YYY of a VALE switch called XXX,
174 both dynamically created if necessary.
175 The string cannot exceed IFNAMSIZ characters, and YYY cannot
176 be the name of any existing OS network interface.
181 indicates the size of the shared memory region,
182 and the number, size and location of all the
184 data structures, which can be accessed by mmapping the memory
185 .Dl char *mem = mmap(0, arg.nr_memsize, fd);
187 Non blocking I/O is done with special
192 on the file descriptor permit blocking I/O.
202 mode, the OS will still believe the interface is up and running.
203 OS-generated packets for that NIC end up into a
205 ring, and another ring is used to send packets into the OS network stack.
208 on the file descriptor removes the binding,
209 and returns the NIC to normal mode (reconnecting the data path
210 to the host stack), or destroys the virtual port.
212 The data structures in the mmapped memory region are detailed in
213 .Xr sys/net/netmap.h ,
214 which is the ultimate reference for the
216 API. The main structures and fields are indicated below:
218 .It Dv struct netmap_if (one per interface)
222 const uint32_t ni_flags; /* properties */
224 const uint32_t ni_tx_rings; /* NIC tx rings */
225 const uint32_t ni_rx_rings; /* NIC rx rings */
226 uint32_t ni_bufs_head; /* head of extra bufs list */
231 Indicates the number of available rings
232 .Pa ( struct netmap_rings )
233 and their position in the mmapped region.
234 The number of tx and rx rings
235 .Pa ( ni_tx_rings , ni_rx_rings )
236 normally depends on the hardware.
237 NICs also have an extra tx/rx ring pair connected to the host stack.
239 can also request additional unbound buffers in the same memory space,
240 to be used as temporary storage for packets.
242 contains the index of the first of these free rings,
243 which are connected in a list (the first uint32_t of each
244 buffer being the index of the next buffer in the list).
245 A 0 indicates the end of the list.
246 .It Dv struct netmap_ring (one per ring)
250 const uint32_t num_slots; /* slots in each ring */
251 const uint32_t nr_buf_size; /* size of each buffer */
253 uint32_t head; /* (u) first buf owned by user */
254 uint32_t cur; /* (u) wakeup position */
255 const uint32_t tail; /* (k) first buf owned by kernel */
258 struct timeval ts; /* (k) time of last rxsync() */
260 struct netmap_slot slot[0]; /* array of slots */
264 Implements transmit and receive rings, with read/write
265 pointers, metadata and and an array of
267 describing the buffers.
268 .It Dv struct netmap_slot (one per buffer)
271 uint32_t buf_idx; /* buffer index */
272 uint16_t len; /* packet length */
273 uint16_t flags; /* buf changed, etc. */
274 uint64_t ptr; /* address for indirect buffers */
278 Describes a packet buffer, which normally is identified by
279 an index and resides in the mmapped region.
280 .It Dv packet buffers
281 Fixed size (normally 2 KB) packet buffers allocated by the kernel.
286 in the mmapped region is indicated by the
288 field in the structure returned by
290 From there, all other objects are reachable through
291 relative references (offsets or indexes).
292 Macros and functions in <net/netmap_user.h>
293 help converting them into actual pointers:
295 .Dl struct netmap_if *nifp = NETMAP_IF(mem, arg.nr_offset);
296 .Dl struct netmap_ring *txr = NETMAP_TXRING(nifp, ring_index);
297 .Dl struct netmap_ring *rxr = NETMAP_RXRING(nifp, ring_index);
299 .Dl char *buf = NETMAP_BUF(ring, buffer_index);
300 .Sh RINGS, BUFFERS AND DATA I/O
302 are circular queues of packets with three indexes/pointers
303 .Va ( head , cur , tail ) ;
304 one slot is always kept empty.
307 should not be assumed to be a power of two.
309 (NOTE: older versions of netmap used head/count format to indicate
310 the content of a ring).
313 is the first slot available to userspace;
317 select/poll will unblock when
323 is the first slot reserved to the kernel.
325 Slot indexes MUST only move forward;
326 for convenience, the function
327 .Dl nm_ring_next(ring, index)
328 returns the next index modulo the ring size.
333 are only modified by the user program;
335 is only modified by the kernel.
336 The kernel only reads/writes the
337 .Vt struct netmap_ring
339 during the execution of a netmap-related system call.
340 The only exception are slots (and buffers) in the range
341 .Va tail\ . . . head-1 ,
342 that are explicitly assigned to the kernel.
345 On transmit rings, after a
347 system call, slots in the range
348 .Va head\ . . . tail-1
349 are available for transmission.
350 User code should fill the slots sequentially
355 past slots ready to transmit.
357 may be moved further ahead if the user code needs
358 more slots before further transmissions (see
359 .Sx SCATTER GATHER I/O ) .
361 At the next NIOCTXSYNC/select()/poll(),
364 are pushed to the port, and
366 may advance if further slots have become available.
367 Below is an example of the evolution of a TX ring:
369 after the syscall, slots between cur and tail are (a)vailable
373 TX [.....aaaaaaaaaaa.............]
375 user creates new packets to (T)ransmit
379 TX [.....TTTTTaaaaaa.............]
381 NIOCTXSYNC/poll()/select() sends packets and reports new slots
385 TX [..........aaaaaaaaaaa........]
388 select() and poll() wlll block if there is no space in the ring, i.e.
389 .Dl ring->cur == ring->tail
390 and return when new slots have become available.
392 High speed applications may want to amortize the cost of system calls
393 by preparing as many packets as possible before issuing them.
395 A transmit ring with pending transmissions has
396 .Dl ring->head != ring->tail + 1 (modulo the ring size).
398 .Va int nm_tx_pending(ring)
399 implements this test.
401 On receive rings, after a
403 system call, the slots in the range
404 .Va head\& . . . tail-1
405 contain received packets.
406 User code should process them and advance
410 past slots it wants to return to the kernel.
412 may be moved further ahead if the user code wants to
413 wait for more packets
414 without returning all the previous slots to the kernel.
416 At the next NIOCRXSYNC/select()/poll(),
419 are returned to the kernel for further receives, and
421 may advance to report new incoming packets.
423 Below is an example of the evolution of an RX ring:
425 after the syscall, there are some (h)eld and some (R)eceived slots
429 RX [..hhhhhhRRRRRRRR..........]
431 user advances head and cur, releasing some slots and holding others
435 RX [..*****hhhRRRRRR...........]
437 NICRXSYNC/poll()/select() recovers slots and reports new packets
441 RX [.......hhhRRRRRRRRRRRR....]
443 .Sh SLOTS AND PACKET BUFFERS
444 Normally, packets should be stored in the netmap-allocated buffers
445 assigned to slots when ports are bound to a file descriptor.
446 One packet is fully contained in a single buffer.
448 The following flags affect slot and buffer processing:
451 it MUST be used when the buf_idx in the slot is changed.
452 This can be used to implement
453 zero-copy forwarding, see
454 .Sx ZERO-COPY FORWARDING .
456 reports when this buffer has been transmitted.
459 notifies transmit completions in batches, hence signals
460 can be delayed indefinitely. This flag helps detecting
461 when packets have been send and a file descriptor can be closed.
463 When a ring is in 'transparent' mode (see
464 .Sx TRANSPARENT MODE ) ,
465 packets marked with this flags are forwarded to the other endpoint
466 at the next system call, thus restoring (in a selective way)
467 the connection between a NIC and the host stack.
469 tells the forwarding code that the SRC MAC address for this
470 packet must not be used in the learning bridge code.
472 indicates that the packet's payload is in a user-supplied buffer,
473 whose user virtual address is in the 'ptr' field of the slot.
474 The size can reach 65535 bytes.
476 This is only supported on the transmit ring of
478 ports, and it helps reducing data copies in the interconnection
481 indicates that the packet continues with subsequent buffers;
482 the last buffer in a packet must have the flag clear.
484 .Sh SCATTER GATHER I/O
485 Packets can span multiple slots if the
487 flag is set in all but the last slot.
488 The maximum length of a chain is 64 buffers.
489 This is normally used with
491 ports when connecting virtual machines, as they generate large
492 TSO segments that are not split unless they reach a physical device.
494 NOTE: The length field always refers to the individual
495 fragment; there is no place with the total length of a packet.
497 On receive rings the macro
499 indicates the remaining number of slots for this packet,
500 including the current one.
501 Slots with a value greater than 1 also have NS_MOREFRAG set.
504 uses two ioctls (NIOCTXSYNC, NIOCRXSYNC)
505 for non-blocking I/O. They take no argument.
506 Two more ioctls (NIOCGINFO, NIOCREGIF) are used
507 to query and configure ports, with the following argument:
510 char nr_name[IFNAMSIZ]; /* (i) port name */
511 uint32_t nr_version; /* (i) API version */
512 uint32_t nr_offset; /* (o) nifp offset in mmap region */
513 uint32_t nr_memsize; /* (o) size of the mmap region */
514 uint32_t nr_tx_slots; /* (i/o) slots in tx rings */
515 uint32_t nr_rx_slots; /* (i/o) slots in rx rings */
516 uint16_t nr_tx_rings; /* (i/o) number of tx rings */
517 uint16_t nr_rx_rings; /* (i/o) number of tx rings */
518 uint16_t nr_ringid; /* (i/o) ring(s) we care about */
519 uint16_t nr_cmd; /* (i) special command */
520 uint16_t nr_arg1; /* (i/o) extra arguments */
521 uint16_t nr_arg2; /* (i/o) extra arguments */
522 uint32_t nr_arg3; /* (i/o) extra arguments */
523 uint32_t nr_flags /* (i/o) open mode */
528 A file descriptor obtained through
530 also supports the ioctl supported by network devices, see
534 returns EINVAL if the named port does not support netmap.
535 Otherwise, it returns 0 and (advisory) information
537 Note that all the information below can change before the
538 interface is actually put in netmap mode.
541 indicates the size of the
543 memory region. NICs in
545 mode all share the same memory region,
548 ports have independent regions for each port.
549 .It Pa nr_tx_slots , nr_rx_slots
550 indicate the size of transmit and receive rings.
551 .It Pa nr_tx_rings , nr_rx_rings
552 indicate the number of transmit
554 Both ring number and sizes may be configured at runtime
555 using interface-specific functions (e.g.
560 binds the port named in
562 to the file descriptor. For a physical device this also switches it into
565 it from the host stack.
566 Multiple file descriptors can be bound to the same port,
567 with proper synchronization left to the user.
569 .Dv NIOCREGIF can also bind a file descriptor to one endpoint of a
571 consisting of two netmap ports with a crossover connection.
572 A netmap pipe share the same memory space of the parent port,
573 and is meant to enable configuration where a master process acts
574 as a dispatcher towards slave processes.
576 To enable this function, the
578 field of the structure can be used as a hint to the kernel to
579 indicate how many pipes we expect to use, and reserve extra space
580 in the memory region.
582 On return, it gives the same info as NIOCGINFO,
587 indicating the identity of the rings controlled through the file
592 selects which rings are controlled through this file descriptor.
595 are indicated below, together with the naming schemes
596 that application libraries (such as the
598 indicated below) can use to indicate the specific set of rings.
599 In the example below, "netmap:foo" is any valid netmap port name.
600 .Bl -tag -width XXXXX
601 .It NR_REG_ALL_NIC "netmap:foo"
602 (default) all hardware ring pairs
603 .It NR_REG_SW "netmap:foo^"
604 the ``host rings'', connecting to the host stack.
605 .It NR_REG_NIC_SW "netmap:foo+"
606 all hardware rings and the host rings
607 .It NR_REG_ONE_NIC "netmap:foo-i"
608 only the i-th hardware ring pair, where the number is in
610 .It NR_REG_PIPE_MASTER "netmap:foo{i"
611 the master side of the netmap pipe whose identifier (i) is in
613 .It NR_REG_PIPE_SLAVE "netmap:foo}i"
614 the slave side of the netmap pipe whose identifier (i) is in
617 The identifier of a pipe must be thought as part of the pipe name,
618 and does not need to be sequential. On return the pipe
619 will only have a single ring pair with index 0,
620 irrespective of the value of i.
627 call pushes out any pending packets on the transmit ring, even if
628 no write events are specified.
629 The feature can be disabled by or-ing
630 .Va NETMAP_NO_TX_POLL
631 to the value written to
633 When this feature is used,
634 packets are transmitted only on
635 .Va ioctl(NIOCTXSYNC)
636 or select()/poll() are called with a write event (POLLOUT/wfdset) or a full ring.
638 When registering a virtual interface that is dynamically created to a
640 switch, we can specify the desired number of rings (1 by default,
641 and currently up to 16) on it using nr_tx_rings and nr_rx_rings fields.
643 tells the hardware of new packets to transmit, and updates the
644 number of slots available for transmission.
646 tells the hardware of consumed packets, and asks for newly available
649 .Sh SELECT, POLL, EPOLL, KQUEUE.
655 file descriptor process rings as indicated in
659 respectively when write (POLLOUT) and read (POLLIN) events are requested.
660 Both block if no slots are available in the ring
661 .Va ( ring->cur == ring->tail ) .
662 Depending on the platform,
668 Packets in transmit rings are normally pushed out
669 (and buffers reclaimed) even without
670 requesting write events. Passing the NETMAP_NO_TX_POLL flag to
672 disables this feature.
673 By default, receive rings are processed only if read
674 events are requested. Passing the NETMAP_DO_RX_POLL flag to
675 .Em NIOCREGIF updates receive rings even without read events.
676 Note that on epoll and kqueue, NETMAP_NO_TX_POLL and NETMAP_DO_RX_POLL
677 only have an effect when some event is posted for the file descriptor.
681 API is supposed to be used directly, both because of its simplicity and
682 for efficient integration with applications.
685 .Va <net/netmap_user.h>
686 header provides a few macros and functions to ease creating
687 a file descriptor and doing I/O with a
689 port. These are loosely modeled after the
691 API, to ease porting of libpcap-based applications to
693 To use these extra functions, programs should
694 .Dl #define NETMAP_WITH_LIBS
696 .Dl #include <net/netmap_user.h>
698 The following functions are available:
699 .Bl -tag -width XXXXX
700 .It Va struct nm_desc * nm_open(const char *ifname, const struct nmreq *req, uint64_t flags, const struct nm_desc *arg)
703 binds a file descriptor to a port.
706 is a port name, in the form "netmap:XXX" for a NIC and "valeXXX:YYY" for a
710 provides the initial values for the argument to the NIOCREGIF ioctl.
711 The nm_flags and nm_ringid values are overwritten by parsing
712 ifname and flags, and other fields can be overridden through
713 the other two arguments.
715 points to a struct nm_desc containing arguments (e.g. from a previously
716 open file descriptor) that should override the defaults.
717 The fields are used as described below
719 can be set to a combination of the following flags:
720 .Va NETMAP_NO_TX_POLL ,
721 .Va NETMAP_DO_RX_POLL
722 (copied into nr_ringid);
723 .Va NM_OPEN_NO_MMAP (if arg points to the same memory region,
724 avoids the mmap and uses the values from it);
725 .Va NM_OPEN_IFNAME (ignores ifname and uses the values in arg);
728 .Va NM_OPEN_ARG3 (uses the fields from arg);
729 .Va NM_OPEN_RING_CFG (uses the ring number and sizes from arg).
731 .It Va int nm_close(struct nm_desc *d)
732 closes the file descriptor, unmaps memory, frees resources.
733 .It Va int nm_inject(struct nm_desc *d, const void *buf, size_t size)
734 similar to pcap_inject(), pushes a packet to a ring, returns the size
735 of the packet is successful, or 0 on error;
736 .It Va int nm_dispatch(struct nm_desc *d, int cnt, nm_cb_t cb, u_char *arg)
737 similar to pcap_dispatch(), applies a callback to incoming packets
738 .It Va u_char * nm_nextpkt(struct nm_desc *d, struct nm_pkthdr *hdr)
739 similar to pcap_next(), fetches the next packet
741 .Sh SUPPORTED DEVICES
743 natively supports the following devices:
761 NICs without native support can still be used in
763 mode through emulation. Performance is inferior to native netmap
764 mode but still significantly higher than sockets, and approaching
765 that of in-kernel solutions such as Linux's
768 Emulation is also available for devices with native netmap support,
769 which can be used for testing or performance comparison.
771 .Va dev.netmap.admode
772 globally controls how netmap mode is implemented.
773 .Sh SYSCTL VARIABLES AND MODULE PARAMETERS
774 Some aspect of the operation of
776 are controlled through sysctl variables on FreeBSD
778 and module parameters on Linux
779 .Em ( /sys/module/netmap_lin/parameters/* ) :
780 .Bl -tag -width indent
781 .It Va dev.netmap.admode: 0
782 Controls the use of native or emulated adapter mode.
783 0 uses the best available option, 1 forces native and
784 fails if not available, 2 forces emulated hence never fails.
785 .It Va dev.netmap.generic_ringsize: 1024
786 Ring size used for emulated netmap mode
787 .It Va dev.netmap.generic_mit: 100000
788 Controls interrupt moderation for emulated mode
789 .It Va dev.netmap.mmap_unreg: 0
790 .It Va dev.netmap.fwd: 0
791 Forces NS_FORWARD mode
792 .It Va dev.netmap.flags: 0
793 .It Va dev.netmap.txsync_retry: 2
794 .It Va dev.netmap.no_pendintr: 1
795 Forces recovery of transmit buffers on system calls
796 .It Va dev.netmap.mitigate: 1
797 Propagates interrupt mitigation to user processes
798 .It Va dev.netmap.no_timestamp: 0
799 Disables the update of the timestamp in the netmap ring
800 .It Va dev.netmap.verbose: 0
801 Verbose kernel messages
802 .It Va dev.netmap.buf_num: 163840
803 .It Va dev.netmap.buf_size: 2048
804 .It Va dev.netmap.ring_num: 200
805 .It Va dev.netmap.ring_size: 36864
806 .It Va dev.netmap.if_num: 100
807 .It Va dev.netmap.if_size: 1024
808 Sizes and number of objects (netmap_if, netmap_ring, buffers)
809 for the global memory region. The only parameter worth modifying is
810 .Va dev.netmap.buf_num
811 as it impacts the total amount of memory used by netmap.
812 .It Va dev.netmap.buf_curr_num: 0
813 .It Va dev.netmap.buf_curr_size: 0
814 .It Va dev.netmap.ring_curr_num: 0
815 .It Va dev.netmap.ring_curr_size: 0
816 .It Va dev.netmap.if_curr_num: 0
817 .It Va dev.netmap.if_curr_size: 0
818 Actual values in use.
819 .It Va dev.netmap.bridge_batch: 1024
820 Batch size used when moving packets across a
822 switch. Values above 64 generally guarantee good
833 to wake up processes when significant events occur, and
837 is used to configure ports and
840 Applications may need to create threads and bind them to
841 specific cores to improve performance, using standard
845 .Xr pthread_setaffinity_np 3
850 comes with a few programs that can be used for testing or
857 .Va tools/tools/netmap/
858 directory in FreeBSD distributions.
861 is a general purpose traffic source/sink.
864 .Dl pkt-gen -i ix0 -f tx -l 60
865 can generate an infinite stream of minimum size packets, and
866 .Dl pkt-gen -i ix0 -f rx
868 Both print traffic statistics, to help monitor
869 how the system performs.
872 has many options can be uses to set packet sizes, addresses,
873 rates, and use multiple send/receive threads and cores.
876 is another test program which interconnects two
878 ports. It can be used for transparent forwarding between
880 .Dl bridge -i ix0 -i ix1
881 or even connect the NIC to the host stack using netmap
882 .Dl bridge -i ix0 -i ix0
883 .Ss USING THE NATIVE API
884 The following code implements a traffic generator
886 .Bd -literal -compact
887 #include <net/netmap_user.h>
891 struct netmap_if *nifp;
892 struct netmap_ring *ring;
896 fd = open("/dev/netmap", O_RDWR);
897 bzero(&nmr, sizeof(nmr));
898 strcpy(nmr.nr_name, "ix0");
899 nmr.nm_version = NETMAP_API;
900 ioctl(fd, NIOCREGIF, &nmr);
901 p = mmap(0, nmr.nr_memsize, fd);
902 nifp = NETMAP_IF(p, nmr.nr_offset);
903 ring = NETMAP_TXRING(nifp, 0);
905 fds.events = POLLOUT;
908 while (!nm_ring_empty(ring)) {
910 buf = NETMAP_BUF(ring, ring->slot[i].buf_index);
911 ... prepare packet in buf ...
912 ring->slot[i].len = ... packet length ...
913 ring->head = ring->cur = nm_ring_next(ring, i);
919 A simple receiver can be implemented using the helper functions
920 .Bd -literal -compact
921 #define NETMAP_WITH_LIBS
922 #include <net/netmap_user.h>
931 d = nm_open("netmap:ix0", NULL, 0, 0);
932 fds.fd = NETMAP_FD(d);
936 while ( (buf = nm_nextpkt(d, &h)) )
937 consume_pkt(buf, h->len);
942 .Ss ZERO-COPY FORWARDING
943 Since physical interfaces share the same memory region,
944 it is possible to do packet forwarding between ports
945 swapping buffers. The buffer from the transmit ring is used
946 to replenish the receive ring:
947 .Bd -literal -compact
949 struct netmap_slot *src, *dst;
951 src = &src_ring->slot[rxr->cur];
952 dst = &dst_ring->slot[txr->cur];
954 dst->buf_idx = src->buf_idx;
956 dst->flags = NS_BUF_CHANGED;
958 src->flags = NS_BUF_CHANGED;
959 rxr->head = rxr->cur = nm_ring_next(rxr, rxr->cur);
960 txr->head = txr->cur = nm_ring_next(txr, txr->cur);
963 .Ss ACCESSING THE HOST STACK
964 The host stack is for all practical purposes just a regular ring pair,
965 which you can access with the netmap API (e.g. with
966 .Dl nm_open("netmap:eth0^", ... ) ;
967 All packets that the host would send to an interface in
969 mode end up into the RX ring, whereas all packets queued to the
970 TX ring are send up to the host stack.
972 A simple way to test the performance of a
974 switch is to attach a sender and a receiver to it,
975 e.g. running the following in two different terminals:
976 .Dl pkt-gen -i vale1:a -f rx # receiver
977 .Dl pkt-gen -i vale1:b -f tx # sender
978 The same example can be used to test netmap pipes, by simply
979 changing port names, e.g.
980 .Dl pkt-gen -i vale:x{3 -f rx # receiver on the master side
981 .Dl pkt-gen -i vale:x}3 -f tx # sender on the slave side
983 The following command attaches an interface and the host stack
985 .Dl vale-ctl -h vale2:em0
988 clients attached to the same switch can now communicate
989 with the network card or the host.
991 .Pa http://info.iet.unipi.it/~luigi/netmap/
993 Luigi Rizzo, Revisiting network I/O APIs: the netmap framework,
994 Communications of the ACM, 55 (3), pp.45-51, March 2012
996 Luigi Rizzo, netmap: a novel framework for fast packet I/O,
997 Usenix ATC'12, June 2012, Boston
999 Luigi Rizzo, Giuseppe Lettieri,
1000 VALE, a switched ethernet for virtual machines,
1001 ACM CoNEXT'12, December 2012, Nice
1003 Luigi Rizzo, Giuseppe Lettieri, Vincenzo Maffione,
1004 Speeding up packet I/O in virtual machines,
1005 ACM/IEEE ANCS'13, October 2013, San Jose
1010 framework has been originally designed and implemented at the
1011 Universita` di Pisa in 2011 by
1013 and further extended with help from
1015 .An Gaetano Catalli ,
1016 .An Giuseppe Lettieri ,
1017 .An Vincenzo Maffione .
1022 have been funded by the European Commission within FP7 Projects
1023 CHANGE (257422) and OPENLAB (287581).
1025 No matter how fast the CPU and OS are,
1026 achieving line rate on 10G and faster interfaces
1027 requires hardware with sufficient performance.
1028 Several NICs are unable to sustain line rate with
1029 small packet sizes. Insufficient PCIe or memory bandwidth
1030 can also cause reduced performance.
1032 Another frequent reason for low performance is the use
1033 of flow control on the link: a slow receiver can limit
1035 Be sure to disable flow control when running high
1038 .Ss SPECIAL NIC FEATURES
1040 is orthogonal to some NIC features such as
1041 multiqueue, schedulers, packet filters.
1043 Multiple transmit and receive rings are supported natively
1044 and can be configured with ordinary OS tools,
1048 device-specific sysctl variables.
1049 The same goes for Receive Packet Steering (RPS)
1050 and filtering of incoming traffic.
1055 .Em checksum offloading , TCP segmentation offloading ,
1056 .Em encryption , VLAN encapsulation/decapsulation ,
1058 When using netmap to exchange packets with the host stack,
1059 make sure to disable these features.