2 .\" Copyright (c) 2012-2016 Intel Corporation
3 .\" All rights reserved.
5 .\" Redistribution and use in source and binary forms, with or without
6 .\" modification, are permitted provided that the following conditions
8 .\" 1. Redistributions of source code must retain the above copyright
9 .\" notice, this list of conditions, and the following disclaimer,
10 .\" without modification.
11 .\" 2. Redistributions in binary form must reproduce at minimum a disclaimer
12 .\" substantially similar to the "NO WARRANTY" disclaimer below
13 .\" ("Disclaimer") and any redistribution must be conditioned upon
14 .\" including a substantially similar Disclaimer requirement for further
15 .\" binary redistribution.
18 .\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
19 .\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
20 .\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTIBILITY AND FITNESS FOR
21 .\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
22 .\" HOLDERS OR CONTRIBUTORS BE LIABLE FOR SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
23 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
24 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
25 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
26 .\" STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
27 .\" IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
28 .\" POSSIBILITY OF SUCH DAMAGES.
30 .\" nvme driver man page.
32 .\" Author: Jim Harris <jimharris@FreeBSD.org>
41 .Nd NVM Express core driver
43 To compile this driver into your kernel,
44 place the following line in your kernel configuration file:
45 .Bd -ragged -offset indent
49 Or, to load the driver as a module at boot, place the following line in
51 .Bd -literal -offset indent
55 Most users will also want to enable
59 to expose NVM Express namespaces as disk devices which can be
61 Note that in NVM Express terms, a namespace is roughly equivalent to a
66 driver provides support for NVM Express (NVMe) controllers, such as:
69 Hardware initialization
71 Per-CPU IO queue pairs
73 API for registering NVMe namespace consumers such as
78 API for submitting NVM commands to namespaces
80 Ioctls for controller and namespace configuration and management
85 driver creates controller device nodes in the format
87 and namespace device nodes in
90 Note that the NVM Express specification starts numbering namespaces at 1,
91 not 0, and this driver follows that convention.
95 will create an I/O queue pair for each CPU, provided enough MSI-X vectors
96 and NVMe queue pairs can be allocated.
97 If not enough vectors or queue
98 pairs are available, nvme(4) will use a smaller number of queue pairs and
99 assign multiple CPUs per queue pair.
101 To force a single I/O queue pair shared by all CPUs, set the following
104 .Bd -literal -offset indent
105 hw.nvme.per_cpu_io_queues=0
108 To assign more than one CPU per I/O queue pair, thereby reducing the number
109 of MSI-X vectors consumed by the device, set the following tunable value in
111 .Bd -literal -offset indent
112 hw.nvme.min_cpus_per_ioq=X
115 To force legacy interrupts for all
117 driver instances, set the following tunable value in
119 .Bd -literal -offset indent
123 Note that use of INTx implies disabling of per-CPU I/O queue pairs.
125 To control maximum amount of system RAM in bytes to use as Host Memory
126 Buffer for capable devices, set the following tunable:
127 .Bd -literal -offset indent
131 The default value is 5% of physical memory size per device.
135 driver is used to provide a disk driver to the system by default.
138 driver can also be used instead.
141 driver performs better with smaller transactions and few TRIM
143 It sends all commands directly to the drive immediately.
146 driver performs better with larger transactions and also collapses
147 TRIM commands giving better performance.
148 It can queue commands to the drive; combine
150 commands into a single trip; and
151 use the CAM I/O scheduler to bias one type of operation over another.
154 driver, set the following tunable value in
156 .Bd -literal -offset indent
160 This value may also be set in the kernel config file with
161 .Bd -literal -offset indent
162 .Cd options NVME_USE_NVD=0
165 When there is an error,
167 prints only the most relevant information about the command by default.
168 To enable dumping of all information about the command, set the following tunable
171 .Bd -literal -offset indent
172 hw.nvme.verbose_cmd_dump=1
176 The following controller-level sysctls are currently implemented:
177 .Bl -tag -width indent
178 .It Va dev.nvme.0.num_cpus_per_ioq
179 (R) Number of CPUs associated with each I/O queue pair.
180 .It Va dev.nvme.0.int_coal_time
181 (R/W) Interrupt coalescing timer period in microseconds.
183 .It Va dev.nvme.0.int_coal_threshold
184 (R/W) Interrupt coalescing threshold in number of command completions.
188 The following queue pair-level sysctls are currently implemented.
189 Admin queue sysctls take the format of dev.nvme.0.adminq and I/O queue sysctls
190 take the format of dev.nvme.0.ioq0.
191 .Bl -tag -width indent
192 .It Va dev.nvme.0.ioq0.num_entries
193 (R) Number of entries in this queue pair's command and completion queue.
194 .It Va dev.nvme.0.ioq0.num_tr
195 (R) Number of nvme_tracker structures currently allocated for this queue pair.
196 .It Va dev.nvme.0.ioq0.num_prp_list
197 (R) Number of nvme_prp_list structures currently allocated for this queue pair.
198 .It Va dev.nvme.0.ioq0.sq_head
199 (R) Current location of the submission queue head pointer as observed by
201 The head pointer is incremented by the controller as it takes commands off
202 of the submission queue.
203 .It Va dev.nvme.0.ioq0.sq_tail
204 (R) Current location of the submission queue tail pointer as observed by
206 The driver increments the tail pointer after writing a command
207 into the submission queue to signal that a new command is ready to be
209 .It Va dev.nvme.0.ioq0.cq_head
210 (R) Current location of the completion queue head pointer as observed by
212 The driver increments the head pointer after finishing
213 with a completion entry that was posted by the controller.
214 .It Va dev.nvme.0.ioq0.num_cmds
215 (R) Number of commands that have been submitted on this queue pair.
216 .It Va dev.nvme.0.ioq0.dump_debug
217 (W) Writing 1 to this sysctl will dump the full contents of the submission
218 and completion queues to the console.
221 In addition to the typical pci attachment, the
223 driver supports attaching to a
226 Intel's Rapid Storage Technology (RST) hides the nvme device
227 behind the AHCI device due to limitations in Windows.
228 However, this effectively hides it from the
231 To work around this limitation,
233 detects that the AHCI device supports RST and when it is enabled.
246 driver first appeared in
252 driver was developed by Intel and originally written by
253 .An Jim Harris Aq Mt jimharris@FreeBSD.org ,
254 with contributions from
258 This man page was written by
259 .An Jim Harris Aq Mt jimharris@FreeBSD.org .