2 .\" Copyright (c) 2012-2016 Intel Corporation
3 .\" All rights reserved.
5 .\" Redistribution and use in source and binary forms, with or without
6 .\" modification, are permitted provided that the following conditions
8 .\" 1. Redistributions of source code must retain the above copyright
9 .\" notice, this list of conditions, and the following disclaimer,
10 .\" without modification.
11 .\" 2. Redistributions in binary form must reproduce at minimum a disclaimer
12 .\" substantially similar to the "NO WARRANTY" disclaimer below
13 .\" ("Disclaimer") and any redistribution must be conditioned upon
14 .\" including a substantially similar Disclaimer requirement for further
15 .\" binary redistribution.
18 .\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
19 .\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
20 .\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTIBILITY AND FITNESS FOR
21 .\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
22 .\" HOLDERS OR CONTRIBUTORS BE LIABLE FOR SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
23 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
24 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
25 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
26 .\" STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
27 .\" IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
28 .\" POSSIBILITY OF SUCH DAMAGES.
30 .\" nvme driver man page.
32 .\" Author: Jim Harris <jimharris@FreeBSD.org>
41 .Nd NVM Express core driver
43 To compile this driver into your kernel,
44 place the following line in your kernel configuration file:
45 .Bd -ragged -offset indent
49 Or, to load the driver as a module at boot, place the following line in
51 .Bd -literal -offset indent
55 Most users will also want to enable
59 to expose NVM Express namespaces as disk devices which can be
61 Note that in NVM Express terms, a namespace is roughly equivalent to a
66 driver provides support for NVM Express (NVMe) controllers, such as:
69 Hardware initialization
71 Per-CPU IO queue pairs
73 API for registering NVMe namespace consumers such as
78 API for submitting NVM commands to namespaces
80 Ioctls for controller and namespace configuration and management
85 driver creates controller device nodes in the format
87 and namespace device nodes in
90 Note that the NVM Express specification starts numbering namespaces at 1,
91 not 0, and this driver follows that convention.
95 will create an I/O queue pair for each CPU, provided enough MSI-X vectors
96 and NVMe queue pairs can be allocated.
97 If not enough vectors or queue
98 pairs are available, nvme(4) will use a smaller number of queue pairs and
99 assign multiple CPUs per queue pair.
101 To force a single I/O queue pair shared by all CPUs, set the following
104 .Bd -literal -offset indent
105 hw.nvme.per_cpu_io_queues=0
108 To assign more than one CPU per I/O queue pair, thereby reducing the number
109 of MSI-X vectors consumed by the device, set the following tunable value in
111 .Bd -literal -offset indent
112 hw.nvme.min_cpus_per_ioq=X
115 To force legacy interrupts for all
117 driver instances, set the following tunable value in
119 .Bd -literal -offset indent
123 Note that use of INTx implies disabling of per-CPU I/O queue pairs.
125 To control maximum amount of system RAM in bytes to use as Host Memory
126 Buffer for capable devices, set the following tunable:
127 .Bd -literal -offset indent
131 The default value is 5% of physical memory size per device.
135 driver is used to provide a disk driver to the system by default.
138 driver can also be used instead.
141 driver performs better with smaller transactions and few TRIM
143 It sends all commands directly to the drive immediately.
146 driver performs better with larger transactions and also collapses
147 TRIM commands giving better performance.
148 It can queue commands to the drive; combine
150 commands into a single trip; and
151 use the CAM I/O scheduler to bias one type of operation over another.
154 driver, set the following tunable value in
156 .Bd -literal -offset indent
160 This value may also be set in the kernel config file with
161 .Bd -literal -offset indent
162 .Cd options NVME_USE_NVD=0
165 When there is an error,
167 prints only the most relevant information about the command by default.
168 To enable dumping of all information about the command, set the following tunable
171 .Bd -literal -offset indent
172 hw.nvme.verbose_cmd_dump=1
175 Prior versions of the driver reset the card twice on boot.
176 This proved to be unnecessary and inefficient, so the driver now resets drive
177 controller only once.
178 The old behavior may be restored in the kernel config file with
179 .Bd -literal -offset indent
180 .Cd options NVME_2X_RESET
184 The following controller-level sysctls are currently implemented:
185 .Bl -tag -width indent
186 .It Va dev.nvme.0.num_cpus_per_ioq
187 (R) Number of CPUs associated with each I/O queue pair.
188 .It Va dev.nvme.0.int_coal_time
189 (R/W) Interrupt coalescing timer period in microseconds.
191 .It Va dev.nvme.0.int_coal_threshold
192 (R/W) Interrupt coalescing threshold in number of command completions.
196 The following queue pair-level sysctls are currently implemented.
197 Admin queue sysctls take the format of dev.nvme.0.adminq and I/O queue sysctls
198 take the format of dev.nvme.0.ioq0.
199 .Bl -tag -width indent
200 .It Va dev.nvme.0.ioq0.num_entries
201 (R) Number of entries in this queue pair's command and completion queue.
202 .It Va dev.nvme.0.ioq0.num_tr
203 (R) Number of nvme_tracker structures currently allocated for this queue pair.
204 .It Va dev.nvme.0.ioq0.num_prp_list
205 (R) Number of nvme_prp_list structures currently allocated for this queue pair.
206 .It Va dev.nvme.0.ioq0.sq_head
207 (R) Current location of the submission queue head pointer as observed by
209 The head pointer is incremented by the controller as it takes commands off
210 of the submission queue.
211 .It Va dev.nvme.0.ioq0.sq_tail
212 (R) Current location of the submission queue tail pointer as observed by
214 The driver increments the tail pointer after writing a command
215 into the submission queue to signal that a new command is ready to be
217 .It Va dev.nvme.0.ioq0.cq_head
218 (R) Current location of the completion queue head pointer as observed by
220 The driver increments the head pointer after finishing
221 with a completion entry that was posted by the controller.
222 .It Va dev.nvme.0.ioq0.num_cmds
223 (R) Number of commands that have been submitted on this queue pair.
224 .It Va dev.nvme.0.ioq0.dump_debug
225 (W) Writing 1 to this sysctl will dump the full contents of the submission
226 and completion queues to the console.
229 In addition to the typical pci attachment, the
231 driver supports attaching to a
234 Intel's Rapid Storage Technology (RST) hides the nvme device
235 behind the AHCI device due to limitations in Windows.
236 However, this effectively hides it from the
239 To work around this limitation,
241 detects that the AHCI device supports RST and when it is enabled.
254 driver first appeared in
260 driver was developed by Intel and originally written by
261 .An Jim Harris Aq Mt jimharris@FreeBSD.org ,
262 with contributions from
266 This man page was written by
267 .An Jim Harris Aq Mt jimharris@FreeBSD.org .