1 .\" Copyright (c) 2011-2014 Matteo Landi, Luigi Rizzo, Universita` di Pisa
2 .\" All rights reserved.
4 .\" Redistribution and use in source and binary forms, with or without
5 .\" modification, are permitted provided that the following conditions
7 .\" 1. Redistributions of source code must retain the above copyright
8 .\" notice, this list of conditions and the following disclaimer.
9 .\" 2. Redistributions in binary form must reproduce the above copyright
10 .\" notice, this list of conditions and the following disclaimer in the
11 .\" documentation and/or other materials provided with the distribution.
13 .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
14 .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
15 .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
16 .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
17 .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
18 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
19 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
20 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
21 .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
22 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
25 .\" This document is derived in part from the enet man page (enet.4)
26 .\" distributed with 4.3BSD Unix.
35 .Nd a framework for fast packet I/O
38 .Nd a fast VirtuAl Local Ethernet using the netmap API
41 .Nd a shared memory packet transport channel
46 is a framework for extremely fast and efficient packet I/O
47 for both userspace and kernel clients.
50 and Linux, and includes
52 a very fast and modular in-kernel software switch/dataplane,
55 a shared memory packet transport channel.
56 All these are accessed interchangeably with the same API.
62 are at least one order of magnitude faster than
63 standard OS mechanisms
64 (sockets, bpf, tun/tap interfaces, native switches, pipes),
65 reaching 14.88 million packets per second (Mpps)
66 with much less than one core on a 10 Gbit NIC,
67 about 20 Mpps per core for VALE ports,
68 and over 100 Mpps for netmap pipes.
70 Userspace clients can dynamically switch NICs into
72 mode and send and receive raw packets through
73 memory mapped buffers.
76 switch instances and ports, and
78 can be created dynamically,
79 providing high speed packet I/O between processes,
80 virtual machines, NICs and the host stack.
83 supports both non-blocking I/O through
85 synchronization and blocking I/O through a file descriptor
86 and standard OS mechanisms such as
95 are implemented by a single kernel module, which also emulates the
97 API over standard drivers for devices without native
100 For best performance,
102 requires explicit support in device drivers.
104 In the rest of this (long) manual page we document
105 various aspects of the
109 architecture, features and usage.
112 supports raw packet I/O through a
114 which can be connected to a physical interface
120 Ports use preallocated circular queues of buffers
122 residing in an mmapped region.
123 There is one ring for each transmit/receive queue of a
125 An additional ring pair connects to the host stack.
127 After binding a file descriptor to a port, a
129 client can send or receive packets in batches through
130 the rings, and possibly implement zero-copy forwarding
133 All NICs operating in
135 mode use the same memory region,
136 accessible to all processes who own
138 file descriptors bound to NICs.
144 by default use separate memory regions,
145 but can be independently configured to share memory.
146 .Sh ENTERING AND EXITING NETMAP MODE
147 The following section describes the system calls to create
155 Simpler, higher level functions are described in section
158 Ports and rings are created and controlled through a file descriptor,
159 created by opening a special device
160 .Dl fd = open("/dev/netmap");
161 and then bound to a specific port with an
162 .Dl ioctl(fd, NIOCREGIF, (struct nmreq *)arg);
165 has multiple modes of operation controlled by the
169 specifies the port name, as follows:
171 .It Dv OS network interface name (e.g. 'em0', 'eth1', ... )
172 the data path of the NIC is disconnected from the host stack,
173 and the file descriptor is bound to the NIC (one or all queues),
174 or to the host stack;
175 .It Dv valeXXX:YYY (arbitrary XXX and YYY)
176 the file descriptor is bound to port YYY of a VALE switch called XXX,
177 both dynamically created if necessary.
178 The string cannot exceed IFNAMSIZ characters, and YYY cannot
179 be the name of any existing OS network interface.
184 indicates the size of the shared memory region,
185 and the number, size and location of all the
187 data structures, which can be accessed by mmapping the memory
188 .Dl char *mem = mmap(0, arg.nr_memsize, fd);
190 Non-blocking I/O is done with special
195 on the file descriptor permit blocking I/O.
205 mode, the OS will still believe the interface is up and running.
206 OS-generated packets for that NIC end up into a
208 ring, and another ring is used to send packets into the OS network stack.
211 on the file descriptor removes the binding,
212 and returns the NIC to normal mode (reconnecting the data path
213 to the host stack), or destroys the virtual port.
215 The data structures in the mmapped memory region are detailed in
216 .In sys/net/netmap.h ,
217 which is the ultimate reference for the
220 The main structures and fields are indicated below:
222 .It Dv struct netmap_if (one per interface)
226 const uint32_t ni_flags; /* properties */
228 const uint32_t ni_tx_rings; /* NIC tx rings */
229 const uint32_t ni_rx_rings; /* NIC rx rings */
230 uint32_t ni_bufs_head; /* head of extra bufs list */
235 Indicates the number of available rings
236 .Pa ( struct netmap_rings )
237 and their position in the mmapped region.
238 The number of tx and rx rings
239 .Pa ( ni_tx_rings , ni_rx_rings )
240 normally depends on the hardware.
241 NICs also have an extra tx/rx ring pair connected to the host stack.
243 can also request additional unbound buffers in the same memory space,
244 to be used as temporary storage for packets.
246 contains the index of the first of these free rings,
247 which are connected in a list (the first uint32_t of each
248 buffer being the index of the next buffer in the list).
251 indicates the end of the list.
252 .It Dv struct netmap_ring (one per ring)
256 const uint32_t num_slots; /* slots in each ring */
257 const uint32_t nr_buf_size; /* size of each buffer */
259 uint32_t head; /* (u) first buf owned by user */
260 uint32_t cur; /* (u) wakeup position */
261 const uint32_t tail; /* (k) first buf owned by kernel */
264 struct timeval ts; /* (k) time of last rxsync() */
266 struct netmap_slot slot[0]; /* array of slots */
270 Implements transmit and receive rings, with read/write
271 pointers, metadata and an array of
273 describing the buffers.
274 .It Dv struct netmap_slot (one per buffer)
277 uint32_t buf_idx; /* buffer index */
278 uint16_t len; /* packet length */
279 uint16_t flags; /* buf changed, etc. */
280 uint64_t ptr; /* address for indirect buffers */
284 Describes a packet buffer, which normally is identified by
285 an index and resides in the mmapped region.
286 .It Dv packet buffers
287 Fixed size (normally 2 KB) packet buffers allocated by the kernel.
292 in the mmapped region is indicated by the
294 field in the structure returned by
296 From there, all other objects are reachable through
297 relative references (offsets or indexes).
298 Macros and functions in
299 .In net/netmap_user.h
300 help converting them into actual pointers:
302 .Dl struct netmap_if *nifp = NETMAP_IF(mem, arg.nr_offset);
303 .Dl struct netmap_ring *txr = NETMAP_TXRING(nifp, ring_index);
304 .Dl struct netmap_ring *rxr = NETMAP_RXRING(nifp, ring_index);
306 .Dl char *buf = NETMAP_BUF(ring, buffer_index);
307 .Sh RINGS, BUFFERS AND DATA I/O
309 are circular queues of packets with three indexes/pointers
310 .Va ( head , cur , tail ) ;
311 one slot is always kept empty.
314 should not be assumed to be a power of two.
316 (NOTE: older versions of netmap used head/count format to indicate
317 the content of a ring).
320 is the first slot available to userspace;
324 select/poll will unblock when
330 is the first slot reserved to the kernel.
335 for convenience, the function
336 .Dl nm_ring_next(ring, index)
337 returns the next index modulo the ring size.
342 are only modified by the user program;
344 is only modified by the kernel.
345 The kernel only reads/writes the
346 .Vt struct netmap_ring
348 during the execution of a netmap-related system call.
349 The only exception are slots (and buffers) in the range
350 .Va tail\ . . . head-1 ,
351 that are explicitly assigned to the kernel.
354 On transmit rings, after a
356 system call, slots in the range
357 .Va head\ . . . tail-1
358 are available for transmission.
359 User code should fill the slots sequentially
364 past slots ready to transmit.
366 may be moved further ahead if the user code needs
367 more slots before further transmissions (see
368 .Sx SCATTER GATHER I/O ) .
370 At the next NIOCTXSYNC/select()/poll(),
373 are pushed to the port, and
375 may advance if further slots have become available.
376 Below is an example of the evolution of a TX ring:
378 after the syscall, slots between cur and tail are (a)vailable
382 TX [.....aaaaaaaaaaa.............]
384 user creates new packets to (T)ransmit
388 TX [.....TTTTTaaaaaa.............]
390 NIOCTXSYNC/poll()/select() sends packets and reports new slots
394 TX [..........aaaaaaaaaaa........]
400 will block if there is no space in the ring, i.e.
401 .Dl ring->cur == ring->tail
402 and return when new slots have become available.
404 High speed applications may want to amortize the cost of system calls
405 by preparing as many packets as possible before issuing them.
407 A transmit ring with pending transmissions has
408 .Dl ring->head != ring->tail + 1 (modulo the ring size).
410 .Va int nm_tx_pending(ring)
411 implements this test.
413 On receive rings, after a
415 system call, the slots in the range
416 .Va head\& . . . tail-1
417 contain received packets.
418 User code should process them and advance
422 past slots it wants to return to the kernel.
424 may be moved further ahead if the user code wants to
425 wait for more packets
426 without returning all the previous slots to the kernel.
428 At the next NIOCRXSYNC/select()/poll(),
431 are returned to the kernel for further receives, and
433 may advance to report new incoming packets.
435 Below is an example of the evolution of an RX ring:
437 after the syscall, there are some (h)eld and some (R)eceived slots
441 RX [..hhhhhhRRRRRRRR..........]
443 user advances head and cur, releasing some slots and holding others
447 RX [..*****hhhRRRRRR...........]
449 NICRXSYNC/poll()/select() recovers slots and reports new packets
453 RX [.......hhhRRRRRRRRRRRR....]
455 .Sh SLOTS AND PACKET BUFFERS
456 Normally, packets should be stored in the netmap-allocated buffers
457 assigned to slots when ports are bound to a file descriptor.
458 One packet is fully contained in a single buffer.
460 The following flags affect slot and buffer processing:
466 in the slot is changed.
467 This can be used to implement
468 zero-copy forwarding, see
469 .Sx ZERO-COPY FORWARDING .
471 reports when this buffer has been transmitted.
474 notifies transmit completions in batches, hence signals
475 can be delayed indefinitely.
476 This flag helps detect
477 when packets have been sent and a file descriptor can be closed.
479 When a ring is in 'transparent' mode (see
480 .Sx TRANSPARENT MODE ) ,
481 packets marked with this flag are forwarded to the other endpoint
482 at the next system call, thus restoring (in a selective way)
483 the connection between a NIC and the host stack.
485 tells the forwarding code that the source MAC address for this
486 packet must not be used in the learning bridge code.
488 indicates that the packet's payload is in a user-supplied buffer
489 whose user virtual address is in the 'ptr' field of the slot.
490 The size can reach 65535 bytes.
492 This is only supported on the transmit ring of
494 ports, and it helps reducing data copies in the interconnection
497 indicates that the packet continues with subsequent buffers;
498 the last buffer in a packet must have the flag clear.
500 .Sh SCATTER GATHER I/O
501 Packets can span multiple slots if the
503 flag is set in all but the last slot.
504 The maximum length of a chain is 64 buffers.
505 This is normally used with
507 ports when connecting virtual machines, as they generate large
508 TSO segments that are not split unless they reach a physical device.
510 NOTE: The length field always refers to the individual
511 fragment; there is no place with the total length of a packet.
513 On receive rings the macro
515 indicates the remaining number of slots for this packet,
516 including the current one.
517 Slots with a value greater than 1 also have NS_MOREFRAG set.
520 uses two ioctls (NIOCTXSYNC, NIOCRXSYNC)
521 for non-blocking I/O.
522 They take no argument.
523 Two more ioctls (NIOCGINFO, NIOCREGIF) are used
524 to query and configure ports, with the following argument:
527 char nr_name[IFNAMSIZ]; /* (i) port name */
528 uint32_t nr_version; /* (i) API version */
529 uint32_t nr_offset; /* (o) nifp offset in mmap region */
530 uint32_t nr_memsize; /* (o) size of the mmap region */
531 uint32_t nr_tx_slots; /* (i/o) slots in tx rings */
532 uint32_t nr_rx_slots; /* (i/o) slots in rx rings */
533 uint16_t nr_tx_rings; /* (i/o) number of tx rings */
534 uint16_t nr_rx_rings; /* (i/o) number of rx rings */
535 uint16_t nr_ringid; /* (i/o) ring(s) we care about */
536 uint16_t nr_cmd; /* (i) special command */
537 uint16_t nr_arg1; /* (i/o) extra arguments */
538 uint16_t nr_arg2; /* (i/o) extra arguments */
539 uint32_t nr_arg3; /* (i/o) extra arguments */
540 uint32_t nr_flags /* (i/o) open mode */
545 A file descriptor obtained through
547 also supports the ioctl supported by network devices, see
551 returns EINVAL if the named port does not support netmap.
552 Otherwise, it returns 0 and (advisory) information
554 Note that all the information below can change before the
555 interface is actually put in netmap mode.
558 indicates the size of the
563 mode all share the same memory region,
566 ports have independent regions for each port.
567 .It Pa nr_tx_slots , nr_rx_slots
568 indicate the size of transmit and receive rings.
569 .It Pa nr_tx_rings , nr_rx_rings
570 indicate the number of transmit
572 Both ring number and sizes may be configured at runtime
573 using interface-specific functions (e.g.
578 binds the port named in
580 to the file descriptor.
581 For a physical device this also switches it into
584 it from the host stack.
585 Multiple file descriptors can be bound to the same port,
586 with proper synchronization left to the user.
588 .Dv NIOCREGIF can also bind a file descriptor to one endpoint of a
590 consisting of two netmap ports with a crossover connection.
591 A netmap pipe share the same memory space of the parent port,
592 and is meant to enable configuration where a master process acts
593 as a dispatcher towards slave processes.
595 To enable this function, the
597 field of the structure can be used as a hint to the kernel to
598 indicate how many pipes we expect to use, and reserve extra space
599 in the memory region.
601 On return, it gives the same info as NIOCGINFO,
606 indicating the identity of the rings controlled through the file
611 selects which rings are controlled through this file descriptor.
614 are indicated below, together with the naming schemes
615 that application libraries (such as the
617 indicated below) can use to indicate the specific set of rings.
618 In the example below, "netmap:foo" is any valid netmap port name.
619 .Bl -tag -width XXXXX
620 .It NR_REG_ALL_NIC "netmap:foo"
621 (default) all hardware ring pairs
622 .It NR_REG_SW "netmap:foo^"
623 the ``host rings'', connecting to the host stack.
624 .It NR_REG_NIC_SW "netmap:foo+"
625 all hardware rings and the host rings
626 .It NR_REG_ONE_NIC "netmap:foo-i"
627 only the i-th hardware ring pair, where the number is in
629 .It NR_REG_PIPE_MASTER "netmap:foo{i"
630 the master side of the netmap pipe whose identifier (i) is in
632 .It NR_REG_PIPE_SLAVE "netmap:foo}i"
633 the slave side of the netmap pipe whose identifier (i) is in
636 The identifier of a pipe must be thought as part of the pipe name,
637 and does not need to be sequential.
639 will only have a single ring pair with index 0,
640 irrespective of the value of
648 call pushes out any pending packets on the transmit ring, even if
649 no write events are specified.
650 The feature can be disabled by or-ing
651 .Va NETMAP_NO_TX_POLL
652 to the value written to
654 When this feature is used,
655 packets are transmitted only on
656 .Va ioctl(NIOCTXSYNC)
657 or select()/poll() are called with a write event (POLLOUT/wfdset) or a full ring.
659 When registering a virtual interface that is dynamically created to a
661 switch, we can specify the desired number of rings (1 by default,
662 and currently up to 16) on it using nr_tx_rings and nr_rx_rings fields.
664 tells the hardware of new packets to transmit, and updates the
665 number of slots available for transmission.
667 tells the hardware of consumed packets, and asks for newly available
670 .Sh SELECT, POLL, EPOLL, KQUEUE.
676 file descriptor process rings as indicated in
680 respectively when write (POLLOUT) and read (POLLIN) events are requested.
681 Both block if no slots are available in the ring
682 .Va ( ring->cur == ring->tail ) .
683 Depending on the platform,
689 Packets in transmit rings are normally pushed out
690 (and buffers reclaimed) even without
691 requesting write events.
693 .Dv NETMAP_NO_TX_POLL
696 disables this feature.
697 By default, receive rings are processed only if read
698 events are requested.
700 .Dv NETMAP_DO_RX_POLL
702 .Em NIOCREGIF updates receive rings even without read events.
703 Note that on epoll and kqueue,
704 .Dv NETMAP_NO_TX_POLL
706 .Dv NETMAP_DO_RX_POLL
707 only have an effect when some event is posted for the file descriptor.
711 API is supposed to be used directly, both because of its simplicity and
712 for efficient integration with applications.
715 .In net/netmap_user.h
716 header provides a few macros and functions to ease creating
717 a file descriptor and doing I/O with a
720 These are loosely modeled after the
722 API, to ease porting of libpcap-based applications to
724 To use these extra functions, programs should
725 .Dl #define NETMAP_WITH_LIBS
727 .Dl #include <net/netmap_user.h>
729 The following functions are available:
730 .Bl -tag -width XXXXX
731 .It Va struct nm_desc * nm_open(const char *ifname, const struct nmreq *req, uint64_t flags, const struct nm_desc *arg)
734 binds a file descriptor to a port.
737 is a port name, in the form "netmap:XXX" for a NIC and "valeXXX:YYY" for a
741 provides the initial values for the argument to the NIOCREGIF ioctl.
742 The nm_flags and nm_ringid values are overwritten by parsing
743 ifname and flags, and other fields can be overridden through
744 the other two arguments.
746 points to a struct nm_desc containing arguments (e.g. from a previously
747 open file descriptor) that should override the defaults.
748 The fields are used as described below
750 can be set to a combination of the following flags:
751 .Va NETMAP_NO_TX_POLL ,
752 .Va NETMAP_DO_RX_POLL
753 (copied into nr_ringid);
754 .Va NM_OPEN_NO_MMAP (if arg points to the same memory region,
755 avoids the mmap and uses the values from it);
756 .Va NM_OPEN_IFNAME (ignores ifname and uses the values in arg);
759 .Va NM_OPEN_ARG3 (uses the fields from arg);
760 .Va NM_OPEN_RING_CFG (uses the ring number and sizes from arg).
762 .It Va int nm_close(struct nm_desc *d)
763 closes the file descriptor, unmaps memory, frees resources.
764 .It Va int nm_inject(struct nm_desc *d, const void *buf, size_t size)
765 similar to pcap_inject(), pushes a packet to a ring, returns the size
766 of the packet is successful, or 0 on error;
767 .It Va int nm_dispatch(struct nm_desc *d, int cnt, nm_cb_t cb, u_char *arg)
768 similar to pcap_dispatch(), applies a callback to incoming packets
769 .It Va u_char * nm_nextpkt(struct nm_desc *d, struct nm_pkthdr *hdr)
770 similar to pcap_next(), fetches the next packet
772 .Sh SUPPORTED DEVICES
774 natively supports the following devices:
792 NICs without native support can still be used in
794 mode through emulation.
795 Performance is inferior to native netmap
796 mode but still significantly higher than sockets, and approaching
797 that of in-kernel solutions such as Linux's
800 Emulation is also available for devices with native netmap support,
801 which can be used for testing or performance comparison.
803 .Va dev.netmap.admode
804 globally controls how netmap mode is implemented.
805 .Sh SYSCTL VARIABLES AND MODULE PARAMETERS
806 Some aspect of the operation of
808 are controlled through sysctl variables on FreeBSD
810 and module parameters on Linux
811 .Em ( /sys/module/netmap_lin/parameters/* ) :
812 .Bl -tag -width indent
813 .It Va dev.netmap.admode: 0
814 Controls the use of native or emulated adapter mode.
815 0 uses the best available option, 1 forces native and
816 fails if not available, 2 forces emulated hence never fails.
817 .It Va dev.netmap.generic_ringsize: 1024
818 Ring size used for emulated netmap mode
819 .It Va dev.netmap.generic_mit: 100000
820 Controls interrupt moderation for emulated mode
821 .It Va dev.netmap.mmap_unreg: 0
822 .It Va dev.netmap.fwd: 0
823 Forces NS_FORWARD mode
824 .It Va dev.netmap.flags: 0
825 .It Va dev.netmap.txsync_retry: 2
826 .It Va dev.netmap.no_pendintr: 1
827 Forces recovery of transmit buffers on system calls
828 .It Va dev.netmap.mitigate: 1
829 Propagates interrupt mitigation to user processes
830 .It Va dev.netmap.no_timestamp: 0
831 Disables the update of the timestamp in the netmap ring
832 .It Va dev.netmap.verbose: 0
833 Verbose kernel messages
834 .It Va dev.netmap.buf_num: 163840
835 .It Va dev.netmap.buf_size: 2048
836 .It Va dev.netmap.ring_num: 200
837 .It Va dev.netmap.ring_size: 36864
838 .It Va dev.netmap.if_num: 100
839 .It Va dev.netmap.if_size: 1024
840 Sizes and number of objects (netmap_if, netmap_ring, buffers)
841 for the global memory region.
842 The only parameter worth modifying is
843 .Va dev.netmap.buf_num
844 as it impacts the total amount of memory used by netmap.
845 .It Va dev.netmap.buf_curr_num: 0
846 .It Va dev.netmap.buf_curr_size: 0
847 .It Va dev.netmap.ring_curr_num: 0
848 .It Va dev.netmap.ring_curr_size: 0
849 .It Va dev.netmap.if_curr_num: 0
850 .It Va dev.netmap.if_curr_size: 0
851 Actual values in use.
852 .It Va dev.netmap.bridge_batch: 1024
853 Batch size used when moving packets across a
856 Values above 64 generally guarantee good
867 to wake up processes when significant events occur, and
871 is used to configure ports and
874 Applications may need to create threads and bind them to
875 specific cores to improve performance, using standard
879 .Xr pthread_setaffinity_np 3
884 comes with a few programs that can be used for testing or
891 .Pa tools/tools/netmap/
897 is a general purpose traffic source/sink.
900 .Dl pkt-gen -i ix0 -f tx -l 60
901 can generate an infinite stream of minimum size packets, and
902 .Dl pkt-gen -i ix0 -f rx
904 Both print traffic statistics, to help monitor
905 how the system performs.
908 has many options can be uses to set packet sizes, addresses,
909 rates, and use multiple send/receive threads and cores.
912 is another test program which interconnects two
915 It can be used for transparent forwarding between
917 .Dl bridge -i ix0 -i ix1
918 or even connect the NIC to the host stack using netmap
919 .Dl bridge -i ix0 -i ix0
920 .Ss USING THE NATIVE API
921 The following code implements a traffic generator
923 .Bd -literal -compact
924 #include <net/netmap_user.h>
928 struct netmap_if *nifp;
929 struct netmap_ring *ring;
933 fd = open("/dev/netmap", O_RDWR);
934 bzero(&nmr, sizeof(nmr));
935 strcpy(nmr.nr_name, "ix0");
936 nmr.nm_version = NETMAP_API;
937 ioctl(fd, NIOCREGIF, &nmr);
938 p = mmap(0, nmr.nr_memsize, fd);
939 nifp = NETMAP_IF(p, nmr.nr_offset);
940 ring = NETMAP_TXRING(nifp, 0);
942 fds.events = POLLOUT;
945 while (!nm_ring_empty(ring)) {
947 buf = NETMAP_BUF(ring, ring->slot[i].buf_index);
948 ... prepare packet in buf ...
949 ring->slot[i].len = ... packet length ...
950 ring->head = ring->cur = nm_ring_next(ring, i);
956 A simple receiver can be implemented using the helper functions
957 .Bd -literal -compact
958 #define NETMAP_WITH_LIBS
959 #include <net/netmap_user.h>
968 d = nm_open("netmap:ix0", NULL, 0, 0);
969 fds.fd = NETMAP_FD(d);
973 while ( (buf = nm_nextpkt(d, &h)) )
974 consume_pkt(buf, h->len);
979 .Ss ZERO-COPY FORWARDING
980 Since physical interfaces share the same memory region,
981 it is possible to do packet forwarding between ports
983 The buffer from the transmit ring is used
984 to replenish the receive ring:
985 .Bd -literal -compact
987 struct netmap_slot *src, *dst;
989 src = &src_ring->slot[rxr->cur];
990 dst = &dst_ring->slot[txr->cur];
992 dst->buf_idx = src->buf_idx;
994 dst->flags = NS_BUF_CHANGED;
996 src->flags = NS_BUF_CHANGED;
997 rxr->head = rxr->cur = nm_ring_next(rxr, rxr->cur);
998 txr->head = txr->cur = nm_ring_next(txr, txr->cur);
1001 .Ss ACCESSING THE HOST STACK
1002 The host stack is for all practical purposes just a regular ring pair,
1003 which you can access with the netmap API (e.g. with
1004 .Dl nm_open("netmap:eth0^", ... ) ;
1005 All packets that the host would send to an interface in
1007 mode end up into the RX ring, whereas all packets queued to the
1008 TX ring are send up to the host stack.
1010 A simple way to test the performance of a
1012 switch is to attach a sender and a receiver to it,
1013 e.g. running the following in two different terminals:
1014 .Dl pkt-gen -i vale1:a -f rx # receiver
1015 .Dl pkt-gen -i vale1:b -f tx # sender
1016 The same example can be used to test netmap pipes, by simply
1017 changing port names, e.g.
1018 .Dl pkt-gen -i vale:x{3 -f rx # receiver on the master side
1019 .Dl pkt-gen -i vale:x}3 -f tx # sender on the slave side
1021 The following command attaches an interface and the host stack
1023 .Dl vale-ctl -h vale2:em0
1026 clients attached to the same switch can now communicate
1027 with the network card or the host.
1029 .Pa http://info.iet.unipi.it/~luigi/netmap/
1031 Luigi Rizzo, Revisiting network I/O APIs: the netmap framework,
1032 Communications of the ACM, 55 (3), pp.45-51, March 2012
1034 Luigi Rizzo, netmap: a novel framework for fast packet I/O,
1035 Usenix ATC'12, June 2012, Boston
1037 Luigi Rizzo, Giuseppe Lettieri,
1038 VALE, a switched ethernet for virtual machines,
1039 ACM CoNEXT'12, December 2012, Nice
1041 Luigi Rizzo, Giuseppe Lettieri, Vincenzo Maffione,
1042 Speeding up packet I/O in virtual machines,
1043 ACM/IEEE ANCS'13, October 2013, San Jose
1048 framework has been originally designed and implemented at the
1049 Universita` di Pisa in 2011 by
1051 and further extended with help from
1053 .An Gaetano Catalli ,
1054 .An Giuseppe Lettieri ,
1056 .An Vincenzo Maffione .
1061 have been funded by the European Commission within FP7 Projects
1062 CHANGE (257422) and OPENLAB (287581).
1064 No matter how fast the CPU and OS are,
1065 achieving line rate on 10G and faster interfaces
1066 requires hardware with sufficient performance.
1067 Several NICs are unable to sustain line rate with
1069 Insufficient PCIe or memory bandwidth
1070 can also cause reduced performance.
1072 Another frequent reason for low performance is the use
1073 of flow control on the link: a slow receiver can limit
1075 Be sure to disable flow control when running high
1077 .Ss SPECIAL NIC FEATURES
1079 is orthogonal to some NIC features such as
1080 multiqueue, schedulers, packet filters.
1082 Multiple transmit and receive rings are supported natively
1083 and can be configured with ordinary OS tools,
1087 device-specific sysctl variables.
1088 The same goes for Receive Packet Steering (RPS)
1089 and filtering of incoming traffic.
1094 .Em checksum offloading , TCP segmentation offloading ,
1095 .Em encryption , VLAN encapsulation/decapsulation ,
1097 When using netmap to exchange packets with the host stack,
1098 make sure to disable these features.