1 .\" SPDX-License-Identifier: BSD-2-Clause
3 .\" Copyright (c) 2015-2022 Amazon.com, Inc. or its affiliates.
4 .\" All rights reserved.
6 .\" Redistribution and use in source and binary forms, with or without
7 .\" modification, are permitted provided that the following conditions
10 .\" 1. Redistributions of source code must retain the above copyright
11 .\" notice, this list of conditions and the following disclaimer.
13 .\" 2. Redistributions in binary form must reproduce the above copyright
14 .\" notice, this list of conditions and the following disclaimer in
15 .\" the documentation and/or other materials provided with the
18 .\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
19 .\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
20 .\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
21 .\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
22 .\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
23 .\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
24 .\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
25 .\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
26 .\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
27 .\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
28 .\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
37 .Nd "FreeBSD kernel driver for Elastic Network Adapter (ENA) family"
39 To compile this driver into the kernel,
40 place the following line in the
41 kernel configuration file:
42 .Bd -ragged -offset indent
46 Alternatively, to load the driver as a
47 module at boot time, place the following line in
49 .Bd -literal -offset indent
53 The ENA is a networking interface designed to make good use of modern CPU
54 features and system architectures.
56 The ENA device exposes a lightweight management interface with a
57 minimal set of memory mapped registers and extendable command set
58 through an Admin Queue.
60 The driver supports a range of ENA devices, is link-speed independent
61 (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has
62 a negotiated and extendable feature set.
64 Some ENA devices support SR-IOV.
65 This driver is used for both the SR-IOV Physical Function (PF) and Virtual
66 Function (VF) devices.
68 The ENA devices enable high speed and low overhead network traffic
69 processing by providing multiple Tx/Rx queue pairs (the maximum number
70 is advertised by the device via the Admin Queue), a dedicated MSI-X
71 interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized
74 When RSS is enabled, each Tx/Rx queue pair is bound to a corresponding
75 CPU core and its NUMA domain.
76 The order of those bindings is based on the RSS bucket mapping.
77 For builds with RSS support disabled, the
78 CPU and NUMA management is left to the kernel.
79 Receive-side scaling (RSS) is supported for multi-core scaling.
83 driver and its corresponding devices implement health
84 monitoring mechanisms such as watchdog, enabling the device and driver
85 to recover in a manner transparent to the application, as well as
88 Some of the ENA devices support a working mode called Low-latency
89 Queue (LLQ), which saves several more microseconds.
93 framework is provided by the
96 Kernel must be built with the DEV_NETMAP option to be able to use this feature.
98 Supported PCI vendor ID/device IDs:
104 1d0f:1ec2 - ENA PF with LLQ support
108 1d0f:ec21 - ENA VF with LLQ support
113 driver's behavior can be changed using run-time or boot-time sysctl
115 The boot-time arguments can be set at the
117 prompt before booting the kernel, or stored in the
119 The run-time arguments can be set using the
124 .Bl -tag -width indent
125 .It Va hw.ena.enable_9k_mbufs
126 Use 9k mbufs for the Rx descriptors.
128 If the node value is set to 1, 9k mbufs will be used for the Rx buffers.
129 If set to 0, page size mbufs will be used instead.
131 Using 9k buffers for Rx can improve Rx throughput, but in low memory conditions
132 it might increase allocation time, as the system has to look for 3 contiguous
134 This can further lead to OS instability, together with ENA driver reset and NVMe
136 If network performance is critical and memory capacity is sufficient, the 9k
138 .It Va hw.ena.force_large_llq_headers
139 Force the driver to use large LLQ headers (224 bytes).
141 If the node value is set to 0, the regular size LLQ header will be used, which
143 In some cases, the packet header can be bigger than this (for example -
144 IPv6 with multiple extensions).
145 In such a situation, the large LLQ headers should be used by setting this node
147 This will take effect only if the device supports both LLQ and large LLQ
149 Otherwise, it will fallback to the no LLQ mode or regular header size.
151 Increasing LLQ header size reduces the size of the Tx queue by half, so it may
152 affect the number of dropped Tx packets.
156 .Bl -tag -width indent
157 .It Va hw.ena.log_level
158 Controls extra logging verbosity of the driver.
160 The higher the logging level, the more logs will be printed out. 0 means all
161 extra logs are disabled and only error logs will be printed out.
162 Default value (2) reports errors, warnings and is verbose about driver
165 The possible flags are:
169 0 - ENA_ERR - Enable driver error messages and ena_com error logs.
171 1 - ENA_WARN - Enable logs for non-critical errors.
173 2 - ENA_INFO - Make the driver more verbose about its actions.
175 3 - ENA_DBG - Enable debug logs.
178 NOTE: In order to enable logging on the Tx/Rx data path, driver must be compiled
179 with ENA_LOG_IO_ENABLE compilation flag.
182 To enable logs for errors and warnings, the following command should be used:
183 .Bd -literal -offset indent
184 sysctl hw.ena.log_level=1
186 .It Va dev.ena.X.io_queues_nb
187 Number of the currently allocated and used IO queues.
188 The default is max_num_io_queues.
189 Controls the number of IO queue pairs (Tx/Rx). As this call has to reallocate
190 the queues, it will reset the interface and restart all the queues - this means
191 that everything, which was currently held in the queue, will be lost, leading to
192 potential packet drops.
194 This call can fail if the system isn't able to provide the driver with enough
196 In that situation, the driver will try to revert the previous number of the IO
198 If this also fails, the device reset will be triggered.
201 To use only 2 Tx and Rx queues for the device ena1, the following command should
203 .Bd -literal -offset indent
204 sysctl dev.ena.1.io_queues_nb=2
206 .It Va dev.ena.X.rx_queue_size
207 Size of the Rx queue.
209 Controls the number of IO descriptors for each Rx queue.
210 The user may want to increase the Rx queue size if they observe a high number of
211 Rx drops in the driver's statistics.
212 For performance reasons, the Rx queue size must be a power of 2.
214 This call can fail if the system isn't able to provide the driver with enough
216 In that situation, the driver will try to revert to the previous number of the
218 If this also fails, the device reset will be triggered.
221 To increase Rx ring size to 8K descriptors for the device ena0, the following
222 command should be used:
223 .Bd -literal -offset indent
224 sysctl dev.ena.0.rx_queue_size=8192
226 .It Va dev.ena.X.buf_ring_size
227 Size of the Tx buffer ring (drbr).
229 Input must be a power of 2.
230 Controls the number of mbufs that can be held in the Tx buffer ring.
231 The drbr is used as a multiple-producer, single-consumer lockless ring for
232 buffering extra mbufs coming from the stack in case the Tx procedure is busy
233 sending the packets, or the Tx ring is full.
234 Increasing the size of the buffer ring may reduce the number of Tx packets being
235 dropped in case of a big Tx burst, which cannot be handled by the IO queue
237 Each Tx queue has its own drbr.
239 It is recommended to keep the drbr with at least the default value, but in case
240 the system lacks the resources, it can be reduced.
241 This call can fail if the system is not able to provide the driver with enough
243 In that situation, the driver will try to revert to the previous number of the
244 drbr and trigger the device reset.
247 To set drbr size for interface ena0 to 2048, the following command should
249 .Bd -literal -offset indent
250 sysctl dev.ena.0.buf_ring_size=2048
252 .It Va dev.ena.X.eni_metrics.sample_interval
253 Interval in seconds for updating ENI metrics.
255 Determines how often (if ever) the ENI metrics should be updated.
256 The ENI metrics are being updated asynchronously in a timer service in order to
257 avoid admin queue overload by sysctl node reading.
258 The value in this node controls the interval between issuing admin commands to
259 the device, which will update the ENI metrics values.
261 If some application is periodically monitoring the eni_metrics, then the ENI
262 metrics interval can be adjusted accordingly.
263 Value 0 turns off the update completely.
264 Value 1 is the minimum interval and is equal to 1 second.
265 The maximum allowed update interval is 1 hour.
268 To update ENI metrics for the device ena1 every 10 seconds, the following
269 command should be used:
270 .Bd -literal -offset indent
271 sysctl dev.ena.1.eni_metrics.sample_interval=10
273 .It Va dev.ena.X.rss.indir_table_size
274 RSS indirection table size.
276 Returns the number of entries in the RSS indirection table.
279 To read the RSS indirection table size, the following command should be used:
280 .Bd -literal -offset indent
281 sysctl dev.ena.0.rss.indir_table_size
283 .It Va dev.ena.X.rss.indir_table
284 RSS indirection table mapping.
285 The default is x:y key-pairs of indir_table_size length.
286 Updates selected indices of the RSS indirection table.
288 The entry string consists of one or more x:y keypairs, where x stands for
289 the table index and y for its new value.
290 Table indices that don't need to be
291 updated can be omitted from the string and will retain their existing values.
293 If an index is entered more than once, the last value is used.
296 To update two selected indices in the RSS indirection table, e.g. setting index
297 0 to queue 5 and then index 5 to queue 0, the following command should be used:
298 .Bd -literal -offset indent
299 sysctl dev.ena.0.rss.indir_table="0:5 5:0"
301 .It Va dev.ena.X.rss.key
303 The default is 40 bytes long randomly generated hash key.
304 Controls the RSS Toeplitz hash algorithm key value.
306 Only available when driver compiled without the kernel side RSS support.
309 To change the RSS hash key value to
311 0x6d, 0x5a, 0x56, 0xda, 0x25, 0x5b, 0x0e, 0xc2,
313 0x41, 0x67, 0x25, 0x3d, 0x43, 0xa3, 0x8f, 0xb0,
315 0xd0, 0xca, 0x2b, 0xcb, 0xae, 0x7b, 0x30, 0xb4,
317 0x77, 0xcb, 0x2d, 0xa3, 0x80, 0x30, 0xf2, 0x0c,
319 0x6a, 0x42, 0xb7, 0x3b, 0xbe, 0xac, 0x01, 0xfa
321 the following command should be used:
322 .Bd -literal -offset indent
323 sysctl dev.ena.0.rss.key=6d5a56da255b0ec24167253d43a38fb0d0ca2bcbae7b30b477cb2da38030f20c6a42b73bbeac01fa
327 .Ss Device initialization phase
329 .It ena%d: failed to init mmio read less
331 Error occurred during initialization of the mmio register read request.
332 .It ena%d: Can not reset device
334 Device could not be reset.
336 Device may not be responding or is already during reset.
337 .It ena%d: device version is too low
339 Version of the controller is too old and it is not supported by the driver.
340 .It ena%d: Invalid dma width value %d
342 The controller is unable to request dma transaction width.
344 Device stopped responding or it demanded invalid value.
345 .It ena%d: Can not initialize ena admin queue with device
347 Initialization of the Admin Queue failed.
349 Device may not be responding or there was a problem with initialization of
351 .It ena%d: Cannot get attribute for ena device rc: %d
353 Failed to get attributes of the device from the controller.
354 .It ena%d: Cannot configure aenq groups rc: %d
356 Errors occurred when trying to configure AENQ groups.
358 .Ss Driver initialization/shutdown phase
360 .It ena%d: PCI resource allocation failed!
361 .It ena%d: failed to pmap registers bar
362 .It ena%d: can not allocate ifnet structure
363 .It ena%d: Error with network interface setup
364 .It ena%d: Failed to enable and set the admin interrupts
365 .It ena%d: Error, MSI-X is already enabled
366 .It ena%d: Failed to enable MSIX, vectors %d rc %d
367 .It ena%d: Not enough number of MSI-X allocated: %d
368 .It ena%d: Error with MSI-X enablement
369 .It ena%d: could not allocate irq vector: %d
370 .It ena%d: unable to allocate bus resource: registers!
371 .It ena%d: unable to allocate bus resource: msix!
373 Resource allocation failed when initializing the device.
375 Driver will not be attached.
376 .It ena%d: ENA device init failed (err: %d)
377 .It ena%d: Cannot initialize device
379 Device initialization failed.
381 Driver will not be attached.
382 .It ena%d: failed to register interrupt handler for irq %ju: %d
384 Error occurred when trying to register Admin Queue interrupt handler.
385 .It ena%d: Cannot setup mgmnt queue intr
387 Error occurred during configuration of the Admin Queue interrupts.
388 .It ena%d: Enable MSI-X failed
390 Configuration of the MSI-X for Admin Queue failed.
392 There could be lack of resources or interrupts could not have been configured.
394 Driver will not be attached.
395 .It ena%d: VLAN is in use, detach first
397 VLANs are being used when trying to detach the driver.
399 VLANs must be detached first and then detach routine have to be called again.
400 .It ena%d: Unmapped RX DMA tag associations
401 .It ena%d: Unmapped TX DMA tag associations
403 Error occurred when trying to destroy RX/TX DMA tag.
404 .It ena%d: Cannot init indirect table
405 .It ena%d: Cannot fill indirect table
406 .It ena%d: Cannot fill hash function
407 .It ena%d: Cannot fill hash control
408 .It ena%d: WARNING: RSS was not properly initialized, it will affect bandwidth
410 Error occurred during initialization of one of RSS resources.
412 The device will work with reduced performance because all RX packets will be
413 passed to queue 0 and there will be no hash information.
414 .It ena%d: LLQ is not supported. Fallback to host mode policy.
415 .It ena%d: Failed to configure the device mode. Fallback to host mode policy.
416 .It ena%d: unable to allocate LLQ bar resource. Fallback to host mode policy.
418 Error occurred during Low-latency Queue mode setup.
420 The device will work, but without the LLQ performance gain.
421 .It ena%d: failed to enable write combining.
423 Error occurred while setting the Write Combining mode, required for the LLQ.
424 .It ena%d: failed to tear down irq: %d
425 .It ena%d: dev has no parent while releasing res for irq: %d
426 Release of the interrupts failed.
428 .Ss Additional diagnostic
430 .It ena%d: Invalid MTU setting. new_mtu: %d max_mtu: %d min mtu: %d
432 Requested MTU value is not supported and will not be set.
433 .It ena%d: Failed to set MTU to %d
435 This message appears when either MTU change feature is not supported, or device
436 communication error has occurred.
437 .It ena%d: Keep alive watchdog timeout.
439 Device stopped responding and will be reset.
440 .It ena%d: Found a Tx that wasn't completed on time, qid %d, index %d.
442 Packet was pushed to the NIC but not sent within given time limit.
444 It may be caused by hang of the IO queue.
445 .It ena%d: The number of lost tx completion is above the threshold (%d > %d). Reset the device
447 If too many Tx weren't completed on time the device is going to be reset.
449 It may be caused by hanged queue or device.
450 .It ena%d: Trigger reset is on
452 Device will be reset.
454 Reset is triggered either by watchdog or if too many TX packets were not
456 .It ena%d: device reset scheduled but trigger_reset is off
458 Reset task has been triggered, but the driver did not request it.
460 Device reset will not be performed.
461 .It ena%d: Device reset failed
463 Error occurred while trying to reset the device.
464 .It ena%d: Cannot initialize device
465 .It ena%d: Error, mac address are different
466 .It ena%d: Error, device max mtu is smaller than ifp MTU
467 .It ena%d: Validation of device parameters failed
468 .It ena%d: Enable MSI-X failed
469 .It ena%d: Failed to create I/O queues
470 .It ena%d: Reset attempt failed. Can not reset the device
472 Error occurred while trying to restore the device after reset.
473 .It ena%d: Device reset completed successfully, Driver info: %s
475 Device has been correctly restored after reset and is ready to use.
476 .It ena%d: Allocation for Tx Queue %u failed
477 .It ena%d: Allocation for Rx Queue %u failed
478 .It ena%d: Unable to create Rx DMA map for buffer %d
479 .It ena%d: Failed to create io TX queue #%d rc: %d
480 .It ena%d: Failed to get TX queue handlers. TX queue num %d rc: %d
481 .It ena%d: Failed to create io RX queue[%d] rc: %d
482 .It ena%d: Failed to get RX queue handlers. RX queue num %d rc: %d
483 .It ena%d: could not allocate irq vector: %d
484 .It ena%d: failed to register interrupt handler for irq %ju: %d
486 IO resources initialization failed.
488 Interface will not be brought up.
489 .It ena%d: LRO[%d] Initialization failed!
491 Initialization of the LRO for the RX ring failed.
492 .It ena%d: failed to alloc buffer for rx queue
493 .It ena%d: failed to add buffer for rx queue %d
494 .It ena%d: refilled rx qid %d with only %d mbufs (from %d)
496 Allocation of resources used on RX path failed.
498 If happened during initialization of the IO queue, the interface will not be
500 .It ena%d: NULL mbuf in rx_info
502 Error occurred while assembling mbuf from descriptors.
503 .It ena%d: tx_info doesn't have valid mbuf
504 .It ena%d: Invalid req_id: %hu
505 .It ena%d: failed to prepare tx bufs
507 Error occurred while preparing a packet for transmission.
508 .It ena%d: ioctl promisc/allmulti
510 IOCTL request for the device to work in promiscuous/allmulti mode.
517 If an issue is identified with the released source code with a supported
518 adapter, please email the specific information related to the issue to
519 .Aq Mt akiyano@amazon.com ,
520 .Aq Mt osamaabb@amazon.com
522 .Aq Mt darinzon@amazon.com .
530 driver first appeared in
535 driver was developed by Amazon and originally written by