1 .\" Copyright (c) 2013 Peter Grehan
2 .\" All rights reserved.
4 .\" Redistribution and use in source and binary forms, with or without
5 .\" modification, are permitted provided that the following conditions
7 .\" 1. Redistributions of source code must retain the above copyright
8 .\" notice, this list of conditions and the following disclaimer.
9 .\" 2. Redistributions in binary form must reproduce the above copyright
10 .\" notice, this list of conditions and the following disclaimer in the
11 .\" documentation and/or other materials provided with the distribution.
13 .\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
14 .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
15 .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
16 .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
17 .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
18 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
19 .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
20 .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
21 .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
22 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
30 .Nd "run a guest operating system inside a virtual machine"
50 .Cm string No | Cm file
59 .Oo Ar bind_address Cm \&: Oc
63 .Op Fl k Ar config_file
67 .Ar lpcdev Op Cm \&, Ar conf
74 .Cm K | Cm k | Cm M | Cm m | Cm G | Cm g | Cm T | Cm t
78 .Op Fl o Ar var Ns Cm = Ns Ar value
79 .Op Fl p Ar vcpu Ns Cm \&: Ns Ar hostcpu
83 .Ar slot Cm \&, Ar emulation Op Cm \&, Ar conf
94 is a hypervisor that runs guest operating systems inside a
96 It can run guests on amd64 and arm64 platforms with suitable hardware support.
98 Parameters such as the number of virtual CPUs, amount of guest memory, and
99 I/O connectivity can be specified with command-line parameters.
102 is typically used with a boot ROM that can load the guest operating system.
103 On arm64 platforms, this is currently required.
104 If not using a boot ROM, the guest operating system must be loaded with
106 or a similar boot loader before running
111 package provides a UEFI firmware that can be used to boot the guest;
113 .Pa u-boot-bhyve-arm64
114 package provides a U-Boot image that can be used to boot the guest.
117 runs until the guest operating system reboots or an unhandled hypervisor
122 The guest's local APIC is configured in xAPIC mode.
123 This option only applies to the amd64 platform.
124 xAPIC mode is the default setting so this option is redundant.
125 It will be deprecated in a future version.
127 Include guest memory in core files.
128 .It Fl c Op Ar setting ...
129 Number of guest virtual CPUs
130 and/or the CPU topology.
131 The default value for each of
140 is not specified then it will be calculated from the other arguments.
141 The topology must be consistent in that the
143 must equal the product of
150 is specified more than once the last one has precedence.
152 The maximum number of virtual CPUs defaults to the number of active
153 physical CPUs in the system available via the
157 The limit can be adjusted via the
161 Destroy the VM on guest initiated power-off.
165 to exit when a guest issues an access to an I/O port that is not emulated.
166 This is intended for debug purposes and only applies to the amd64 platform.
167 .It Fl f Ar name Ns Cm \&, Ns Oo Cm string Ns No | Ns Cm file Ns Oc Ns Cm \&= Ns Ar data
170 to the fw_cfg interface.
173 is specified, the fw_cfg file contains the string as data.
176 is specified, bhyve reads the file and adds the file content as fw_cfg data.
180 .Oo Ar bind_address Cm \&: Oc
184 Start a debug server that uses the GDB protocol to export guest state to a
186 An IPv4 TCP socket will be bound to the supplied
190 to listen for debugger connections.
191 Only a single debugger may be attached to the debug server at a time.
192 If the option begins with
195 will pause execution at the first instruction waiting for a debugger to attach.
197 Yield the virtual CPU thread when a HLT instruction is detected.
198 If this option is not specified, virtual CPUs will use 100% of a host CPU.
199 This option applies only to the amd64 platform.
201 Print help message and exit.
202 .It Fl k Ar config_file
203 Set configuration variables from a simple, key-value config file.
204 Each line of the config file is expected to consist of a config variable
208 No spaces are permitted between the variable name, equals sign, or
210 Blank lines and lines starting with
217 Specify the keyboard layout.
218 The value that can be specified sets the file name in
219 .Ar /usr/share/bhyve/kbdlayout .
220 This specification only works when loaded with UEFI mode for VNC.
221 When using a VNC client that supports QEMU Extended Key Event Message (e.g.
222 TigerVNC), this option isn't needed.
223 When using a VNC client that doesn't support QEMU Extended Key Event Message
224 (e.g. tightVNC), the layout defaults to the US keyboard unless specified
227 Print a list of supported LPC devices.
228 .It Fl l Ar lpcdev Ns Op Cm \&, Ns Ar conf
229 Allow devices behind the LPC PCI-ISA bridge to be configured.
230 The only supported devices are the TTY-class devices
231 .Cm com1 , com2 , com3 ,
240 type and the debug/test device
243 The possible values for the
245 argument are listed in the
249 This option applies only to the amd64 platform.
250 On arm64, the console and boot ROM devices are configured using the
255 .Fl m Ar memsize Ns Oo
257 .Cm K | k | M | m | G | g | T | t
261 Set the guest physical memory size.
262 This must be the same size that was given to
265 The size argument may be suffixed with one of
269 (either upper or lower case)
270 to indicate a multiple of kilobytes, megabytes, gigabytes, or terabytes.
271 If no suffix is given, the value is assumed to be in megabytes.
274 .It Fl o Ar var Ns Cm = Ns Ar value
275 Set the configuration variable
281 for configuration options.
283 Force the guest virtual CPU to exit when a PAUSE instruction is detected.
284 This option applies only to the amd64 platform.
285 .It Fl p Ar vcpu Ns Cm \& : Ns Ar hostcpu
286 Pin guest's virtual CPU
290 Host CPUs and guest virtual CPUs are numbered starting from 0.
293 option is required for every guest vCPU to be pinned.
294 To map a 4 vCPU guest to host CPUs 12-15:
296 -p 0:12 -p 1:13 -p 2:14 -p 3:15
299 Resume a guest from a snapshot.
300 The guest memory contents are restored from
302 and the guest device and vCPU state are restored from the file
303 .Dq Ar file Ns .kern .
305 Note that the current snapshot file format requires that the
306 configuration of devices in the new VM match the VM from which the
307 snapshot was taken by specifying the same
312 The count of vCPUs and memory configuration are read from the snapshot.
316 Print a list of supported PCI devices.
317 .It Fl s Ar slot Ns Cm \&, Ns Ar emulation Ns Op Cm \&, Ns Ar conf
318 Configure a virtual PCI slot and function.
321 provides PCI bus emulation and virtual devices that can be attached to
323 There are 32 available slots, with the option of providing up to 8 functions
328 can be specified in one of the following formats:
335 .Ar pcislot Cm \&: Ar function
339 .Ar bus Cm \&: Ar pcislot Cm \&: Ar function
352 If not specified, the
355 If not specified, the
362 can be one of the following:
363 .Bl -tag -width "amd_hostbridge"
365 A simple host bridge.
366 This is usually configured at slot 0, and is required by most guest
368 .It Cm amd_hostbridge
369 Emulation identical to
371 using a PCI vendor ID of AMD.
373 PCI pass-through device.
375 Virtio network interface.
377 Virtio block storage interface.
379 Virtio SCSI interface.
381 Virtio 9p (VirtFS) interface.
383 Virtio RNG interface.
384 .It Cm virtio-console
385 Virtio console interface, which exposes multiple ports
386 to the guest in the form of simple char devices for simple IO
387 between the guest and host userspaces.
389 Virtio input interface.
391 AHCI controller attached to arbitrary devices.
393 AHCI controller attached to an ATAPI CD/DVD.
395 AHCI controller attached to a SATA hard drive.
397 Intel e82545 network interface.
399 PCI 16550 serial device.
401 LPC PCI-ISA bridge with COM1, COM2, COM3, and COM4 16550 serial ports,
403 optionally, a fwcfg type and the debug/test device.
404 The LPC bridge emulation can only be configured on bus 0.
406 Raw framebuffer device attached to VNC server.
408 eXtensible Host Controller Interface (xHCI) USB controller.
410 NVM Express (NVMe) controller.
412 High Definition Audio Controller.
415 The optional parameter
417 describes the backend for device emulations.
420 is not specified, the device emulation has no backend and can be
421 considered unconnected.
423 Network device backends:
429 .Op Cm \&,mac= Ar xx:xx:xx:xx:xx:xx
435 .Op Cm \&,mac= Ar xx:xx:xx:xx:xx:xx
440 .Cm netgraph,path= Ar ADDRESS Cm \&,peerhook= Ar HOOK
441 .Op Cm \&,socket= Ar NAME
442 .Op Cm \&,hook= Ar HOOK
443 .Op Cm \&,mac= Ar xx:xx:xx:xx:xx:xx
448 .Cm slirp,hostfwd= Ar proto : Ar hostaddr : Ar hostport - Ar guestaddr : Ar guestport
455 is not specified, the MAC address is derived from a fixed OUI and the
456 remaining bytes from an MD5 hash of the slot and function numbers and
459 The MAC address is an ASCII string in
467 parameter can be specified to inform the guest about the largest MTU
468 that should be allowed, expressed in bytes.
476 parameters must be specified to set the destination node and corresponding hook.
477 The optional parameters
481 may be used to set the
483 node name and source hook.
493 The slirp backend can be used to provide a NATed network to the guest.
494 This backend has poor performance but does not require any network
495 configuration on the host system.
501 option takes a 5-tuple describing how connections from the host are to be
502 forwarded to the guest.
503 Multiple rules can be specified, separated by semicolons.
504 Note that semicolons must be escaped or quoted to prevent the shell from
507 Block storage device backends:
511 .Ar /filename Op Cm \&, Ar block-device-options
513 .Ar /dev/xxx Op Cm \&, Ar block-device-options
518 .Ar block-device-options
528 Force the file to be opened read-only.
529 .It Cm sectorsize= Ns Ar logical Ns Oo Cm \&/ Ns Ar physical Oc
530 Specify the logical and physical sector sizes of the emulated disk.
531 The physical sector size is optional and is equal to the logical sector size
532 if not explicitly specified.
534 Disable emulation of guest trim requests via
537 .It Li bootindex= Ns Ar index
538 Add the device to the bootorder at
540 A fwcfg file is used to specify the bootorder.
541 The guest firmware may ignore or doesn't support this fwcfg file.
542 In that case, this feature doesn't work as expected.
545 SCSI device backends:
549 .Pa /dev/cam/ctl Oo Ar pp Cm \&. Ar vp Oc Oo Cm \&, Ar scsi-device-options Oc
554 .Ar scsi-device-options
557 .It Cm iid= Ns Ar IID
558 Initiator ID to use when sending requests to specified CTL port.
559 The default value is 0.
560 .It Li bootindex= Ns Ar index
561 Add the device to the bootorder at
563 A fwcfg file is used to specify the bootorder.
564 The guest firmware may ignore or doesn't support this fwcfg file.
565 In that case, this feature doesn't work as expected.
572 .Ar sharename Cm = Ar /path/to/share Op Cm \&, Ar 9p-device-options
577 .Ar 9p-device-options
581 Expose the share in read-only mode.
587 Connect the serial port to the standard input and output of
592 Use the host TTY device for serial port I/O.
597 .It Ar type Ns \&, Ns Ar path Ns Op Cm \&, Ns Ar tpm-device-options
598 Emulate a TPM device.
602 .Ar tpm-device-options
605 .It Cm version= Ns Ar version
606 Version of the TPM device according to the TCG specification.
611 Boot ROM device backends:
613 .It Ar romfile Ns Op Cm \&, Ns Ar varfile
616 in the guest address space reserved for boot firmware.
619 is provided, that file is also mapped in the boot firmware guest
620 address space, and any modifications the guest makes will be saved
627 The fwcfg interface is used to pass information such as the CPU count
628 or ACPI tables to the guest firmware.
633 Due to backward compatibility reasons,
635 is the default option.
638 is used, bhyve's fwctl interface is used.
639 It currently reports only the CPU count to the guest firmware.
642 option uses QEMU's fwcfg interface.
643 This interface is widely used and allows user-defined information to
644 be passed to the guest.
645 It is used for passing the CPU count, ACPI tables, a boot order and
646 many other things to the guest.
647 Some operating systems such as Fedora CoreOS can be configured by
648 qemu's fwcfg interface as well.
651 Pass-through device backends:
655 .Cm ppt Ar N Oo , Ar passthru-device-options Oc
657 .Ns Ar bus Cm \&/ Ar slot Cm \&/ Ar function
658 .Op , Ar passthru-device-options
660 .Cm pci Ar bus Cm : Ar slot Cm : Ns Ar function
661 .Op , Ar passthru-device-options
665 Connect to a PCI device on the host either named ppt
667 or at the selector described by
675 .Ar passthru-device-options
678 .It Cm rom= Ns Ar romfile
681 as option ROM to the PCI device.
682 The ROM will be loaded by firmware and should be capable of
683 initializing the device.
684 .It Li bootindex= Ns Ar index
685 Add the device to the bootorder at
687 A fwcfg file is used to specify the bootorder.
688 The guest firmware may ignore or doesn't support this fwcfg file.
689 In that case, this feature doesn't work as expected.
692 Guest memory must be wired using the
694 option when a pass-through device is configured.
696 The host device must have been reserved at boot-time using the
698 loader variable as described in
704 Specifies the type of the TPM device.
710 .It Cm version= Ns Ar version
713 of the emulated TPM device according to the TCG specification.
721 Virtio console device backends:
725 .Cm port1= Ns Ar /path/to/port1.sock Ns Op Cm ,port Ns Ar N Cm \&= Ns Ar /path/to/port2.sock No \~ Ar ...
729 A maximum of 16 ports per device can be created.
730 Every port is named and corresponds to a Unix domain socket created by
733 accepts at most one connection per port at a time.
738 Due to lack of destructors in
740 sockets on the filesystem must be cleaned up manually after
744 There is no way to use the
746 feature, nor the console port
749 Emergency write is advertised, but no-op at present.
752 Virtio input device backends:
754 .It Ar /dev/input/eventX
756 .Ar /dev/input/eventX
757 to guest by VirtIO Input Interface.
760 Framebuffer devices backends:
764 .Op Cm rfb= Ar ip-and-port
767 .Op Cm ,vga= Ar vgaconf
769 .Op Cm ,password= Ar password
773 Configuration options are defined as follows:
775 .It Cm rfb= Ns Ar ip-and-port Pq or Cm tcp= Ns Ar ip-and-port
776 An IP address and a port VNC should listen on.
777 There are two formats:
787 .Cm \&[ Ar IPv6%zone Cm \&] Cm \&: Ar port
791 The default is to listen on localhost IPv4 address and default VNC port 5900.
792 An IPv6 address must be enclosed in square brackets and may contain an
793 optional zone identifier.
794 .It Cm w= Ns Ar width No and Cm h= Ns Ar height
795 A display resolution, width and height, respectively.
796 If not specified, a default resolution of 1024x768 pixels will be used.
797 Minimal supported resolution is 640x480 pixels,
798 and maximum is 3840x2160 pixels.
799 .It Cm vga= Ns Ar vgaconf
800 Possible values for this option are
806 PCI graphics cards have a dual personality in that they are
807 standard PCI devices with BAR addressing, but may also
808 implicitly decode legacy VGA I/O space
811 .Pq 64KB at Ad 0xA0000 .
814 option should be used for guests that attempt to issue BIOS calls which result
815 in I/O port queries, and fail to boot if I/O decode is disabled.
819 option should be used along with the CSM BIOS capability in UEFI
820 to boot traditional BIOS guests that require the legacy VGA I/O and
821 memory regions to be available.
825 option should be used for the UEFI guests that assume that
826 VGA adapter is present if they detect the I/O ports.
827 An example of such a guest is
835 .Pq Lk https://wiki.freebsd.org/bhyve
836 for configuration notes of particular guests.
840 to only boot upon the initiation of a VNC connection, simplifying the
841 installation of operating systems that require immediate keyboard input.
842 This can be removed for post-installation use.
843 .It Cm password= Ns Ar password
844 This type of authentication is known to be cryptographically weak and is not
845 intended for use on untrusted networks.
846 Many implementations will want to use stronger security, such as running
847 the session over an encrypted channel provided by IPsec or SSH.
850 xHCI USB device backends:
853 A USB tablet device which provides precise cursor synchronization
857 NVMe device backends:
864 .Op Cm ,ioslots= Ar #
872 Configuration options are defined as follows:
875 Accepted device paths are:
880 .Cm ram= Ns Ar size_in_MiB .
882 Max number of queues.
884 Max elements in each queue.
886 Max number of concurrent I/O requests.
888 Sector size (defaults to blockif sector size).
890 Serial number with maximum 20 characters.
892 IEEE Extended Unique Identifier (8 byte value).
894 DataSet Management support.
895 Supported values are:
901 AHCI device backends:
905 .Op Oo Cm hd\&: | cd\&: Oc Ar path
906 .Op Cm ,nmrr= Ar nmrr
913 Configuration options are defined as follows:
916 Nominal Media Rotation Rate, known as RPM.
917 Value 1 will indicate device as Solid State Disk.
918 Default value is 0, not report.
920 Serial Number with maximum 20 characters.
922 Revision Number with maximum 8 characters.
924 Model Number with maximum 40 characters.
927 HD Audio device backends:
931 .Op Cm play= Ar playback
932 .Op Cm ,rec= Ar recording
936 Configuration options are defined as follows:
939 Playback device, typically
942 Recording device, typically
946 Set the universally unique identifier
948 in the guest's System Management BIOS System Information structure.
949 By default a UUID is generated from the host's hostname and
954 Force virtio PCI device emulations to use MSI interrupts instead of MSI-X
957 Ignore accesses to unimplemented Model Specific Registers (MSRs).
958 This is intended for debug purposes.
960 The guest's local APIC is configured in x2APIC mode.
961 This option applies only to the amd64 platform.
963 Disable MPtable generation.
964 This option applies only to the amd64 platform.
966 Alphanumeric name of the guest.
967 This should be the same as that created by
970 .Sh CONFIGURATION VARIABLES
972 uses an internal tree of configuration variables to describe global and
977 it parses command line options (including config files) in the order given
979 Each command line option sets one or more configuration variables.
983 option creates a new tree node for a PCI device and sets one or more variables
984 under that node including the device model and device model-specific variables.
985 Variables may be set multiple times during this parsing stage with the final
986 value overriding previous values.
988 Once all of the command line options have been processed,
989 the configuration values are frozen.
991 then uses the value of configuration values to initialize device models
994 More details on configuration variables can be found in
997 The current debug server provides limited support for debuggers.
999 Each virtual CPU is exposed to the debugger as a thread.
1001 General purpose registers can be queried for each virtual CPU, but other
1002 registers such as floating-point and system registers cannot be queried.
1004 Memory (including memory mapped I/O regions) can be read and written
1006 Memory operations use virtual addresses that are resolved to physical
1007 addresses via the current virtual CPU's active address translation.
1009 The running guest can be interrupted by the debugger at any time
1010 .Pq for example, by pressing Ctrl-C in the debugger .
1012 Single stepping is only supported on Intel CPUs supporting the MTRAP VM exit.
1014 Breakpoints are supported on Intel CPUs that support single stepping.
1015 Note that continuing from a breakpoint while interrupts are enabled in the
1016 guest may not work as expected due to timer interrupts firing while single
1017 stepping over the breakpoint.
1020 deals with the following signals:
1022 .Bl -tag -width SIGTERM -compact
1024 Trigger ACPI poweroff for a VM
1027 Exit status indicates how the VM was terminated:
1029 .Bl -tag -width indent -compact
1039 exited due to an error
1042 If not using a boot ROM, the guest operating system must have been loaded with
1044 or a similar boot loader before
1047 Otherwise, the boot loader is not needed.
1049 To run a virtual machine with 1GB of memory, two virtual CPUs, a virtio
1050 block device backed by the
1052 filesystem image, and a serial port for the console:
1053 .Bd -literal -offset indent
1054 bhyve -c 2 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,/my/image \\
1055 -l com1,stdio -H -P -m 1G vm1
1058 To do the same on arm64:
1059 .Bd -literal -offset indent
1061 bhyve -c 2 -s 0,hostbridge -s 1,virtio-blk,/my/image -o console=stdio \\
1062 -o bootrom=/usr/local/share/u-boot/u-boot-bhyve-arm64/u-boot.bin -m 1G vm1
1064 Run a 24GB single-CPU virtual machine with three network ports, one of which
1065 has a MAC address specified:
1066 .Bd -literal -offset indent
1067 bhyve -s 0,hostbridge -s 1,lpc -s 2:0,virtio-net,tap0 \\
1068 -s 2:1,virtio-net,tap1 \\
1069 -s 2:2,virtio-net,tap2,mac=00:be:fa:76:45:00 \\
1070 -s 3,virtio-blk,/my/image -l com1,stdio \\
1074 Run an 8GB quad-CPU virtual machine with 8 AHCI SATA disks, an AHCI ATAPI
1075 CD-ROM, a single virtio network port, an AMD hostbridge, and the console
1076 port connected to an
1079 .Bd -literal -offset indent
1081 -s 0,amd_hostbridge -s 1,lpc \\
1082 -s 1:0,ahci,hd:/images/disk.1,hd:/images/disk.2,\\
1083 hd:/images/disk.3,hd:/images/disk.4,\\
1084 hd:/images/disk.5,hd:/images/disk.6,\\
1085 hd:/images/disk.7,hd:/images/disk.8,\\
1086 cd:/images/install.iso \\
1087 -s 3,virtio-net,tap0 \\
1088 -l com1,/dev/nmdm0A \\
1092 Run a UEFI virtual machine with a display resolution of 800 by 600 pixels
1093 that can be accessed via VNC at: 0.0.0.0:5900.
1094 .Bd -literal -offset indent
1095 bhyve -c 2 -m 4G -w -H \\
1097 -s 3,ahci-cd,/path/to/uefi-OS-install.iso \\
1098 -s 4,ahci-hd,disk.img \\
1099 -s 5,virtio-net,tap0 \\
1100 -s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait \\
1101 -s 30,xhci,tablet \\
1102 -s 31,lpc -l com1,stdio \\
1103 -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \\
1107 Run a UEFI virtual machine with a VNC display that is bound to all IPv6
1108 addresses on port 5900.
1109 .Bd -literal -offset indent
1110 bhyve -c 2 -m 4G -w -H \\
1112 -s 4,ahci-hd,disk.img \\
1113 -s 5,virtio-net,tap0 \\
1114 -s 29,fbuf,tcp=[::]:5900,w=800,h=600 \\
1115 -s 30,xhci,tablet \\
1116 -s 31,lpc -l com1,stdio \\
1117 -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \\
1121 Run a UEFI virtual machine with a VARS file to save EFI variables.
1124 will write guest modifications to the given VARS file.
1125 Be sure to create a per-guest copy of the template VARS file from
1127 .Bd -literal -offset indent
1128 bhyve -c 2 -m 4g -w -H \\
1130 -s 31,lpc -l com1,stdio \\
1131 -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd,BHYVE_UEFI_VARS.fd
1140 .Xr bhyve_config 5 ,
1147 .%B 64 and IA-32 Architectures Software Developer’s Manual
1155 .An Neel Natu Aq Mt neel@freebsd.org
1156 .An Peter Grehan Aq Mt grehan@freebsd.org
projects / FreeBSD / FreeBSD.git / blob