Warner Losh [Sun, 8 Jul 2018 07:42:58 +0000 (07:42 +0000)]
Stop using ../zfs/libzfs.h but instead use libzfs.h.
While ../zfs/libzfs.h mostly works, there are a few situations where
it does not. Eliminate the problem by using plain libzfs.h, like we do
for ufs support. This fixes the weird cases, and is easier to
understand. It also follows the general style convetion of avoiding
../ in #includes.
Warner Losh [Sun, 8 Jul 2018 07:42:49 +0000 (07:42 +0000)]
Move ZFS files into libsa
Move the libzfs stuff into libsa. There's no need for it to be a
separate library. The separate library adds to the issues of build
ordering that we see from time to time. Move the filesystem support
into libsa, like all the other filesystem support rather than making
zfs the odd-duck out.
Rick Macklem [Sat, 7 Jul 2018 19:27:49 +0000 (19:27 +0000)]
Fix handling of the hybrid DS case for a pNFS server.
After the addition of the "#mds_path" suffix for a DS specification on the
"-p" nfsd option, it is possible to have a mix of DSs assigned to an MDS
file system and DSs that store files for all DSs. This is what I referred
to as "hybrid" above.
At first, I didn't think this hybrid case would be useful, but I now believe
that some system administrators may fine it useful.
This patch modifies the file storage assignment algorithm so that it
makes the "#mds_path" DSs take priority and the all file systems DSs
are now only used for MDS file systems with no "#mds_path" DS servers.
This only affects the pNFS server for this "hybrid" case.
Ian Lepore [Sat, 7 Jul 2018 19:10:00 +0000 (19:10 +0000)]
Add support to the imx watchdog for the FDT "timeout-sec" property, by
automatically initializing the watchdog using the given value. Also,
attach at BUS_PASS_TIMER to extend watchdog protection to more of the
kernel init process.
Ian Lepore [Sat, 7 Jul 2018 19:03:38 +0000 (19:03 +0000)]
Correctly calculate the value to put in the imx wdog countdown register.
The correct value is seconds*2-1. The code was using just seconds*2, which
led to being off by a half-second -- usually not a big deal, except when the
value was the max (128) it overflowed so zero would get written to the
countdown register, which equates to a timeout of a half second.
Sean Bruno [Sat, 7 Jul 2018 13:35:06 +0000 (13:35 +0000)]
struct ifmediareq *ifmrp is only used in the COMPAT_FREEBSD32 parts of
ifioctl(). Move it inside the proper #ifdef. This was throwing a valid
"Assigned but unused" warning with gcc.
Michael Tuexen [Sat, 7 Jul 2018 12:28:16 +0000 (12:28 +0000)]
Allow alternate TCP stack to populate the TCP FO client cookie
cache.
Without this patch, TCP FO could be used when using alternate
TCP stack, but only existing entires in the TCP client cookie
cache could be used. This cache was not populated by connections
using alternate TCP stacks.
Ed Schouten [Sat, 7 Jul 2018 11:53:39 +0000 (11:53 +0000)]
Allow the use of slashes in process names of RFC 3164 formatted messages.
Tools such as Postfix use slashes in process names for hierarchy
(postfix/qmgr). By allowing these slashes, syslogd is able to extract
the process name and process ID nicely, so that they can be stored in
RFC 5424 message fields.
Michael Tuexen [Sat, 7 Jul 2018 11:18:26 +0000 (11:18 +0000)]
When initializing the TCP FO client cookie cache, take into account
whether the TCP FO support is enabled or not for the client side.
The code in tcp_fastopen_init() implicitly assumed that the sysctl
variable V_tcp_fastopen_client_enable was initialized to 0. This
was initially true, but was changed in r335610, which unmasked this
bug.
Thanks to Pieter de Goeje for reporting the issue on freebsd-net@
ofw_iicbus already has attachments on iichb. Rather than adding an explicit
attachment onto opal_i2c, simply change the exposed name of the OPAL i2c bus
to 'iichb'.
Save a call to pmap_remove() if entry cannot have any pages mapped.
Due to the way rtld creates mappings for the shared objects, each dso
causes unmap of at least three guard map entries. For instance, in
the buildworld load, this change reduces the amount of pmap_remove()
calls by 1/5.
Profiled by: alc
Reviewed by: alc, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D16148
Ed Maste [Fri, 6 Jul 2018 19:44:49 +0000 (19:44 +0000)]
Fix arm64 linuxulator clone() argument order
Linux/arm64 is CLONE_BACKWARDS - i.e., "Architecture has tls passed as
the 4th argument of clone(2), not the 5th one."
The linux clone() syscall has four different permutations of argument
order, depending on architecture - see the #ifdef CONFIG_CLONE_BACKWARDS
maze in Linux's kernel/fork.c.
Rick Macklem [Fri, 6 Jul 2018 19:18:45 +0000 (19:18 +0000)]
Change the pNFS server so that it does not disable a mirrored DS for
an NFSERR_STALE error reported via a LayoutReturn.
The current FreeBSD client can generate these errors for an operational
DS while doing a recovery of a mirror after a mirrored DS has been repaired.
I am not sure why these errors occur, but my best current guess is a race
between the Layout Recall issued by the kernel code run from pnfsdscopymr(8)
and a Read operation on the DS for the file bing copied.
The errors are not fatal, since the client falls back on doing I/O through
the MDS, which can do the I/O successfully as a proxy. (The fact that the
MDS can do this indicates that the file does still exist on the functioning
DS.)
This change only affects the pNFS server and only when a client does a
LayoutReturn with the NFSERR_STALE error report.
Change prison_add_vfs() to the more generic prison_add_allow(), which
can add any dynamic allow.* or allow.*.* parameter. Also keep
prison_add_vfs() as a wrapper.
The dynamic environment was being initialized at SI_SUB_KMEM, SI_ORDER_ANY.
I added the hint-merging at SI_SUB_KMEM, SI_ORDER_ANY as well in r335998 -
this can only work by coincidence.
Re-do both to operate at SI_SUB_KMEM + 1, SI_ORDER_FIRST and SI_ORDER_SECOND
respectively to be safe. It's sufficiently obfuscated away as to when in
SU_SUB_KMEM malloc will be available, and the dynamic environment cannot be
relied upon there anyways since it's initialized at SI_ORDER_ANY.
Sean Bruno [Fri, 6 Jul 2018 13:22:44 +0000 (13:22 +0000)]
r336028 changed next_msg to a char * from char [] of fixed size. Change
2nd argument of vsnprintf() to get the strlen of next_msg so that the
appropriate size is used.
Found with gcc.
/usr.bin/top/display.c: In function 'new_message':
/usr.bin/top/display.c:963:31: error:
argument to 'sizeof' in 'vsnprintf' call is the same expression as the
destination; did you mean to provide an explicit length?
[-Werror=sizeof-pointer-memaccess]
vsnprintf(next_msg, sizeof(next_msg), msgfmt, args);
Save a call to pmap_remove() if entry cannot have any pages mapped.
Due to the way rtld creates mappings for the shared objects, each dso
causes unmap of at least three guard map entries. For instance, in
the buildworld load, this change reduces the amount of pmap_remove()
calls by 1/5.
Profiled by: alc
Reviewed by: alc, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D16148
r336019 introduced ${SRCTOP}/sys to the include paths in order to pull in a
new sys/{c,}nv.h. This is wrong, because the build tree's ABI isn't
guaranteed to match what's running on the host system.
Fix instead by removing -I${SRCTOP}/sys and installing the libnv headers
with `make -C lib/libnv includes`... this may or may not get re-worked in
the future so that a userland lib isn't installing includes from sys/.
Matt Macy [Fri, 6 Jul 2018 10:10:00 +0000 (10:10 +0000)]
counter(9): unbreak amd64 following r336020
Apply temporary fix to counter until daylight hours.
The fact that the assembly for counter_u64_add relied on the sizeof(struct pcpu) was
the basis for the otherwise arbitrary offset never came up in D15933.
critical_{enter,exit} is now inline so the only real added overhead is the
added (mostly false) conditional branch in exit.
Matt Macy [Fri, 6 Jul 2018 02:06:03 +0000 (02:06 +0000)]
Back pcpu zone with domain correct pages
- Change pcpu zone consumers to use a stride size of PAGE_SIZE.
(defined as UMA_PCPU_ALLOC_SIZE to make future identification easier)
- Allocate page from the correct domain for a given cpu.
- Don't initialize pc_domain to non-zero value if NUMA is not defined
There are some misconceptions surrounding this field. It is the
_VM_ NUMA domain and should only ever correspond to valid domain
values as understood by the VM.
The former slab size of sizeof(struct pcpu) was somewhat arbitrary.
The new value is PAGE_SIZE because that's the smallest granularity
which the VM can allocate a slab for a given domain. If you have
fewer than PAGE_SIZE/8 counters on your system there will be some
memory wasted, but this is obviously something where you want the
cache line to be coming from the correct domain.
Reviewed by: jeff
Sponsored by: Limelight Networks
Differential Revision: https://reviews.freebsd.org/D15933
config(8): De-dupe hint/env vars within a single file
r335653 flipped the order in which hints/env files are concatenated to match
the order in which vars are processed by the kernel. This is the other
hammer to drop.
Use nv(9) to de-dupe entries within a single `hint` or `env` file, using the
latest value specified for a key. This leaves some duplicates if a variable
is specified in multiple hint/env files or via `envvar` in a kernel config,
but the reversed order of concatenation (from r335653) makes this a
non-issue as the latest-specified version will be seen first.
This change also silently rewrote hint bits to use the same sanitization
process that ian@ wrote for r335642. To the kernel, hints and env vars are
basically the same thing through early boot, then get merged into the
dynamic environment once kmem becomes available and the dynamic environment
is created. They should be subjected to the same restrictions.
libnv has been added to -legacy for the time being to support the build of
config(8) with the new cnvlist API.
Sean Eric Fagan [Thu, 5 Jul 2018 22:56:13 +0000 (22:56 +0000)]
This exposes ZFS user and group quotas via the normal
quatactl(2) mechanism. (Read-only at this point, however.)
In particular, this is to allow rpc.rquotad query quotas
for NFS mounts, allowing users to see their quotas on the
hosts using the datasets.
The changes specifically:
* Add new RPC entry points for querying quotas.
* Changes the library routines to allow non-UFS quotas.
* Changes rquotad to check for quotas on mounted filesystems,
rather than being limited to entries in /etc/fstab
* Lastly, adds a VFS entry-point for ZFS to query quotas.
Note that this makes one unavoidable behavioural change: if quotas
are enabled, then they can be queried, as opposed to the current
method of checking for quotas being specified in fstab. (With
ZFS, if there are user or group quotas, they're used, always.)
config(8): De-dupe hint/env vars within a single file
r335653 flipped the order in which hints/env files are concatenated to match
the order in which vars are processed by the kernel. This is the other
hammer to drop.
Use nv(9) to de-dupe entries within a single `hint` or `env` file, using the
latest value specified for a key. This leaves some duplicates if a variable
is specified in multiple hint/env files or via `envvar` in a kernel config,
but the reversed order of concatenation (from r335653) makes this a
non-issue as the latest-specified version will be seen first.
This change also silently rewrote hint bits to use the same sanitization
process that ian@ wrote for r335642. To the kernel, hints and env vars are
basically the same thing through early boot, then get merged into the
dynamic environment once kmem becomes available and the dynamic environment
is created. They should be subjected to the same restrictions.
It is possible that a fictitious unmanaged userspace mapping of
superpage is created on x86, e.g. by pmap_object_init_pt(), with the
physical address outside the vm_page_array[] coverage.
Noted and reviewed by: alc
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D16085
Andrew Turner [Thu, 5 Jul 2018 17:13:37 +0000 (17:13 +0000)]
Create a new macro for static DPCPU data.
On arm64 (and possible other architectures) we are unable to use static
DPCPU data in kernel modules. This is because the compiler will generate
PC-relative accesses, however the runtime-linker expects to be able to
relocate these.
In preparation to fix this create two macros depending on if the data is
global or static.
Sean Bruno [Thu, 5 Jul 2018 17:07:23 +0000 (17:07 +0000)]
Make ZSTD a real option via ZSTDIO.
It looks like the intent was to allow ZSTD support to be
compiled into the kernel with options ZSTDIO. But it doesn't look
like that was ever implemented or I'm missing how to do it.
I did a cursory audit of kernel config files and made a decision to
enable ZSTDIO in riscv GENERIC and mips MALTA configurations. All other
kernel configurations already had this option in their kernel configs
but they didn't do anything useful as the feature was declared as
"standard" prior to this.
Reviewed by: cem allanjude
Differential Revision: https://reviews.freebsd.org/D16007
Copyout(9) on 4/4 i386 needs correct vm_page_array[].
On the 4/4 i386, copyout(9) may need to call pmap_extract_and_hold()
on arbitrary userspace mapping. If the mapping is backed by the
non-managed cdev pager or by the sg pager, on dense configs we might
access arbitrary element of vm_page_array[], in particular, not
corresponding to a page from the memory segment. Initialize such pages
as fictitious with the corresponding physical address.
Reported by: bde
Reviewed by: alc, markj (previous version)
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D16085
In x86 pmap_extract_and_hold(), there is no need to recalculate the
physical address, which is readily available after sucessfull
vm_page_pa_tryrelock().
Noted and reviewed by: alc
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D16085
kern_environment: use any provided environments, evict hintmode/envmode
At the moment, hintmode and envmode are used to indicate whether static
hints or static env have been provided in the kernel config(5) and the
static versions are mutually exclusive with loader(8)-provided environment.
hintmode *can* be reconfigured later to pull from the dynamic environment,
thus taking advantage of the loader(8) or post-kmem environment setting.
This changeset fixes both problems at once to move us from a semi-confusing
state to a consistent state: if an environment file, hints file, or
loader(8) environment are provided, we use them in a well-known order of
precedence:
Once the dynamic environment is setup this becomes a moot point. The
loader(8) and static environments are merged (respecting the above order of
precedence), and the static hints are merged in on an as-needed basis after
the dynamic environment has been setup.
Hints lookup are changed to respect all of the above. Before the dynamic
environment is setup, lookups use the above-mentioned order and fallback to
the next environment if a matching hint is not found. Once the dynamic
environment is setup, that is used on its own since it captures all of the
above information plus any dynamic kenv settings that came up later in boot.
The following tangentially related changes were made to res_find:
- A hintp cookie is now passed in so that related searches continue using
the chain of environments (or dynamic environment) without relying on
global state
- All three environments will be searched if they actually have valid hints
to use, rather than just choosing the first environment that actually had
a hint and rolling with that only
The hintmode sysctl has been ripped out. static_{env,hints}.disabled are
still honored and will disable their respective environments from being used
for hint lookups and from being merged into the dynamic environment, as
expected.
In x86 pmap_extract_and_hold(), there is no need to recalculate the
physical address, which is readily available after sucessfull
vm_page_pa_tryrelock().
Noted and reviewed by: alc
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D16085
kern_environment: use any provided environments, evict hintmode/envmode
At the moment, hintmode and envmode are used to indicate whether static
hints or static env have been provided in the kernel config(5) and the
static versions are mutually exclusive with loader(8)-provided environment.
hintmode *can* be reconfigured later to pull from the dynamic environment,
thus taking advantage of the loader(8) or post-kmem environment setting.
This changeset fixes both problems at once to move us from a semi-confusing
state to a consistent state: if an environment file, hints file, or
loader(8) environment are provided, we use them in a well-known order of
precedence:
Once the dynamic environment is setup this becomes a moot point. The
loader(8) and static environments are merged (respecting the above order of
precedence), and the static hints are merged in on an as-needed basis after
the dynamic environment has been setup.
Hints lookup are changed to respect all of the above. Before the dynamic
environment is setup, lookups use the above-mentioned order and fallback to
the next environment if a matching hint is not found. Once the dynamic
environment is setup, that is used on its own since it captures all of the
above information plus any dynamic kenv settings that came up later in boot.
The following tangentially related changes were made to res_find:
- A hintp cookie is now passed in so that related searches continue using
the chain of environments (or dynamic environment) without relying on
global state
- All three environments will be searched if they actually have valid hints
to use, rather than just choosing the first environment that actually had
a hint and rolling with that only
The hintmode sysctl has been ripped out. static_{env,hints}.disabled are
still honored and will disable their respective environments from being used
for hint lookups and from being merged into the dynamic environment, as
expected.
With the introduction of reapers and reaplists in r275800,
proc0 and init are setup as a circular dependency.
create_init() calls fork1() which calls do_fork(). There the
newproc (initproc) is setup with a reaper of proc0 who's reaper
points to itself. The newproc (initproc) is then put on its
reaper's (proc0) p_reaplist (initproc is a descendants of proc0
for proc0 to reap). Upon return to create_init(), proc0 is
added to initproc's p_reaplist (which would mean proc0 is a
descendant of init, for init to reap). This creates a
circular dependency which eventually leads to LIST corruptions
when trying to kill init and a proc0.
For the base system we never really hit this case during reboot.
The problem only became visible after adding more virtual process
spaces which could go away cleanly (work existing in an experimental
branch).
Reviewed by: kib
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D15924
Ian Lepore [Thu, 5 Jul 2018 16:00:58 +0000 (16:00 +0000)]
Detach all children before beginning to tear down the hardware, instead of
doing it last. Also, remove the local tracking of whether usb's busdma
memory allocation got done, because it's safe to call the free_all
function even if it wasn't.
Ian Lepore [Thu, 5 Jul 2018 15:34:16 +0000 (15:34 +0000)]
Remove a test and early-out which just can't possibly be right. It causes
detach() to do nothing if attach() succeeded, which is the opposite of
what's needed. Also, move device_delete_children() from the end to the
beginning of detach(), so that children won't be trying to make use of the
hardware we're in the process of shutting down.
Ian Lepore [Thu, 5 Jul 2018 14:09:48 +0000 (14:09 +0000)]
Fix an out-of-bounds array access... the irq data for teardown is in two
arrays, as elements 0 and 1 of one array and elements 1 and 2 of the other.
Run the loop 0..1 instead of 1..2 and use named constants to offset into
one of the arrays.
Brooks Davis [Thu, 5 Jul 2018 13:13:48 +0000 (13:13 +0000)]
Make struct xinpcb and friends word-size independent.
Replace size_t members with ksize_t (uint64_t) and pointer members
(never used as pointers in userspace, but instead as unique
idenitifiers) with kvaddr_t (uint64_t). This makes the structs
identical between 32-bit and 64-bit ABIs.
On 64-bit bit systems, the ABI is maintained. On 32-bit systems,
this is an ABI breaking change. The ABI of most of these structs
was previously broken in r315662. This also imposes a small API
change on userspace consumers who must handle kernel pointers
becoming virtual addresses.
The initial work on bhyve NVMe device emulation was done by the GSoC student
Shunsuke Mie and was heavily modified in performan, functionality and
guest support by Leon Dang.
Tested with guest OS: FreeBSD Head, Linux Fedora fc27, Ubuntu 18.04,
OpenSuse 15.0, Windows Server 2016 Datacenter.
Tested with all accepted device paths: Real nvme, zdev and also with ram.
Tested on: AMD Ryzen Threadripper 1950X 16-Core Processor and
Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz.
Alan Cox [Thu, 5 Jul 2018 02:04:18 +0000 (02:04 +0000)]
As of r335784, if pmap_enter() replaces a managed mapping by an unmanaged
mapping, then it leaks the unlinked PV entry. This change eliminates that
leak, freeing the PV entry.
Rick Macklem [Wed, 4 Jul 2018 19:46:26 +0000 (19:46 +0000)]
Fix the pNFS server so that it handles the "#mds_path" check for mirrors.
The recently added feature of the pNFS server will set an fsid for the
MDS file system to define the file system a DS should store files for.
For a case where a DS handling all file systems has failed, it was possible
for the code to check for a mirror with a specified fs, even though
nfsdev_mdsisset was 0, possibly causing a false successful check for a mirror.
This patch adds a check for nfsdev_mdsisset != 0 to avoid this.
It only affects the pNFS server for a rare case. Found via code inspection.
projects / FreeBSD / FreeBSD.git / log