Matthew Macy [Mon, 2 Dec 2019 18:08:27 +0000 (10:08 -0800)]
Move zfs_cmd_t copyin/copyout to platform code
FreeBSD needs to cope with multiple version of the zfs_cmd_t
structure. Allowing the platform code to pre and post
process the cmd structure makes it possible to work with
legacy tooling.
Reviewed-by: Jorgen Lundman <lundman@lundman.net> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9624
Matthew Macy [Sat, 30 Nov 2019 23:45:07 +0000 (15:45 -0800)]
Restructure nvlist_nv_alloc to work on FreeBSD
KM_PUSHPAGE is an Illumosism - On FreeBSD it's
aliased to the same malloc flag as KM_SLEEP.
The compiler naturally rejects multiple case
statements with the same value. This is effectively
a no-op since all callers pass a specific KM_* flag.
Reviewed-by: Jorgen Lundman <lundman@lundman.net> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9643
InsanePrawn [Fri, 22 Nov 2019 17:53:51 +0000 (18:53 +0100)]
Fix encryption logic in systemd mount generator
Previously the generator would skip a dataset if it wasn't mountable by
'zfs mount -a' (legacy/none mountpoint, canmount off/noauto). This also
skipped the generation of key-load units for such datasets, breaking
the dependency handling for mountable child datasets.
Reviewed-by: Antonio Russo <antonio.e.russo@gmail.com> Reviewed-by: Richard Laager <rlaager@wiktel.com> Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
Closes #9611
Paul Zuchowski [Wed, 27 Nov 2019 18:45:56 +0000 (13:45 -0500)]
Implement -A (ignore ASSERTs) for zdb
The command line switch -A (ignore ASSERTs) has always been available
in zdb but was never connected up to the correct global variable.
There are times when you need zdb to ignore asserts and keep dumping
out whatever information it can get despite the ASSERT(s) failing.
It was always intended to be part of zdb but was incomplete.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Paul Zuchowski <pzuchowski@datto.com>
Closes #9610
Brian Behlendorf [Wed, 27 Nov 2019 18:35:49 +0000 (10:35 -0800)]
Remove zfs_vdev_elevator module option
As described in commit f81d5ef6 the zfs_vdev_elevator module
option is being removed. Users who require this functionality
should update their systems to set the disk scheduler using a
udev rule.
Reviewed-by: Richard Laager <rlaager@wiktel.com> Reviewed-by: loli10K <ezomori.nozomu@gmail.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #8664
Closes #9417
Closes #9609
jwpoduska [Wed, 27 Nov 2019 18:15:01 +0000 (13:15 -0500)]
Prevent unnecessary resilver restarts
If a device is participating in an active resilver, then it will have a
non-empty DTL. Operations like vdev_{open,reopen,probe}() can cause the
resilver to be restarted (or deferred to be restarted later), which is
unnecessary if the DTL is still covered by the current scan range. This
is similar to the logic in vdev_dtl_should_excise() where the DTL can
only be excised if it's max txg is in the resilvered range.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: John Gallagher <john.gallagher@delphix.com> Reviewed-by: Kjeld Schouten <kjeld@schouten-lebbing.nl> Signed-off-by: John Poduska <jpoduska@datto.com>
Issue #840
Closes #9155
Closes #9378
Closes #9551
Closes #9588
Paul Zuchowski [Wed, 27 Nov 2019 18:08:18 +0000 (13:08 -0500)]
Add display of checksums to zdb -R
The function zdb_read_block (zdb -R) was always intended to have a :c
flag which would read the DVA and length supplied by the user, and
display the checksum. Since we don't know which checksum goes with
the data, we should calculate and display them all.
For each checksum in the table, read in the data at the supplied
DVA:length, calculate the checksum, and display it. Update the man
page and create a zfs test for the new feature.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Kjeld Schouten <kjeld@schouten-lebbing.nl> Signed-off-by: Paul Zuchowski <pzuchowski@datto.com>
Closes #9607
The changes in commit 41e1aa2a / PR #9583 introduced a regression on
tmpfile_001_pos: fsetxattr() on a O_TMPFILE file descriptor started
to fail with errno ENODATA:
Matthew Macy [Thu, 21 Nov 2019 17:32:57 +0000 (09:32 -0800)]
Add zfs_file_* interface, remove vnodes
Provide a common zfs_file_* interface which can be implemented on all
platforms to perform normal file access from either the kernel module
or the libzpool library.
This allows all non-portable vnode_t usage in the common code to be
replaced by the new portable zfs_file_t. The associated vnode and
kobj compatibility functions, types, and macros have been removed
from the SPL. Moving forward, vnodes should only be used in platform
specific code when provided by the native operating system.
Reviewed-by: Sean Eric Fagan <sef@ixsystems.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Igor Kozhukhov <igor@dilos.org> Reviewed-by: Jorgen Lundman <lundman@lundman.net> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9556
Brian Behlendorf [Thu, 21 Nov 2019 01:26:32 +0000 (17:26 -0800)]
ZTS: tst.terminate_by_signal increase test threshold
The tst.terminate_by_signal test case may occasionally fail when
running in a less consistent virtual environment. For all observed
failures the process was terminated correctly but it took longer than
expected resulting in too many snapshot being created.
To minimize the likelyhood of this occuring increase the threshold
from 50 to 90 snapshots. The larger limit will still verifiy that
the channel program was correctly terminated early.
Reviewed-by: Don Brady <don.brady@delphix.com> Reviewed-by: Reviewed-by: Kjeld Schouten <kjeld@schouten-lebbing.nl> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #9601
George Melikov [Wed, 20 Nov 2019 00:23:27 +0000 (03:23 +0300)]
ZTS: Casenorm fix unicode interpretation
Use `printf` to properly interpret unicode characters.
Illumos uses a utility called `zlook` to allow additional flags to be
provided to readdir and lookup for testing. This functionality could
be ported to Linux, but even without it several of the tests can be
enabled by instead using the standard `test` command.
Additional, work is required to enable the remaining test cases.
Reviewed-by: Igor Kozhukhov <igor@dilos.org> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: George Melikov <mail@gmelikov.ru>
Issue #7633
Closes #8812
InsanePrawn [Tue, 19 Nov 2019 00:44:28 +0000 (01:44 +0100)]
Remove requirement for -d 1 for zfs list and zfs get with bookmarks
df58307 removed the need to specify -d 1 when zfs list and zfs get are
called with -t snapshot on a datset. This commit extends the same
behaviour to -t bookmark.
This commit also introduces the 'snap' shorthand for snapshots from
zfs list to zfs get.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Tom Caputi <tcaputi@datto.com> Reviewed-by: Kjeld Schouten <kjeld@schouten-lebbing.nl> Signed-off-by: InsanePrawn <insane.prawny@gmail.com>
Closes #9589
Reinstate the zpl_revalidate() functionality to resolve a regression
where dentries for open files during a rollback are not invalidated.
The unrelated functionality for automatically unmounting .zfs/snapshots
was not reverted. Nor was the addition of shrink_dcache_sb() to the
zfs_resume_fs() function.
This issue was not immediately caught by the CI because the test case
intended to catch it was included in the list of ZTS tests which may
occasionally fail for unrelated reasons. Remove all of the rollback
tests from this list to help identify the frequency of any spurious
failures.
The rollback_003_pos.ksh test case exposes a real issue with the
long standing code which needs to be investigated. Regardless,
it has been enable with a small workaround in the test case itself.
Reviewed-by: Matt Ahrens <matt@delphix.com> Reviewed-by: Pavel Snajdr <snajpa@snajpa.net> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #9587
Closes #9592
If zp->z_unlinked is set, we're working with a znode that has been
marked for deletion. If that's the case, we can skip the "goto again"
loop and return ENOENT, as the znode should not be discovered.
Reviewed-by: Richard Yao <ryao@gentoo.org> Reviewed-by: Matt Ahrens <mahrens@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Heitor Alves de Siqueira <halves@canonical.com>
Closes #9583
InsanePrawn [Fri, 15 Nov 2019 17:52:11 +0000 (18:52 +0100)]
Remove inappropiate error message suggesting to use '-r'
Removes an incorrect error message from libzfs that suggests applying
'-r' when a zfs subcommand is called with a filesystem path while
expecting either a snapshot or bookmark path.
loli10K [Wed, 13 Nov 2019 18:19:06 +0000 (19:19 +0100)]
Prevent NULL pointer dereference in blkg_tryget() on EL8 kernels
blkg_tryget() as shipped in EL8 kernels does not seem to handle NULL
@blkg as input; this is different from its mainline counterpart where
NULL is accepted. To prevent dereferencing a NULL pointer when dealing
with block devices which do not set a root_blkg on the request queue
perform the NULL check in vdev_bio_associate_blkg().
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Kjeld Schouten <kjeld@schouten-lebbing.nl> Reviewed-by: Tony Hutter <hutter2@llnl.gov> Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #9546
Closes #9577
Check for __GFP_RECLAIM instead of GFP_KERNEL because zfs modifies
IO and FS flags which breaks the check for GFP_KERNEL.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Matt Ahrens <matt@delphix.com> Reviewed-by: Sebastian Gottschall <s.gottschall@dd-wrt.com> Signed-off-by: Michael Niewöhner <foss@mniewoehner.de>
Closes #9034
This adds a new KMC_KVMEM flag was added to enforce use of the
kvmalloc allocator in kmem_cache_create even for large blocks, which
may also increase performance in some specific cases (e.g. zstd), too.
Default to KVMEM instead of VMEM in spl_kmem_cache_create.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Matt Ahrens <matt@delphix.com> Signed-off-by: Sebastian Gottschall <s.gottschall@dd-wrt.com> Signed-off-by: Michael Niewöhner <foss@mniewoehner.de>
Closes #9034
Make use of kvmalloc if available and fix vmem_alloc implementation
This patch implements use of kvmalloc for GFP_KERNEL allocations, which
may increase performance if the allocator is able to allocate physical
memory, if kvmalloc is available as a public kernel interface (since
v4.12). Otherwise it will simply fall back to virtual memory (vmalloc).
Also fix vmem_alloc implementation which can lead to slow allocations
since the first attempt with kmalloc does not make use of the noretry
flag but tells the linux kernel to retry several times before it fails.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Matt Ahrens <matt@delphix.com> Signed-off-by: Sebastian Gottschall <s.gottschall@dd-wrt.com> Signed-off-by: Michael Niewöhner <foss@mniewoehner.de>
Closes #9034
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Matt Ahrens <matt@delphix.com> Signed-off-by: Sebastian Gottschall <s.gottschall@dd-wrt.com> Signed-off-by: Michael Niewöhner <foss@mniewoehner.de>
Closes #9034
Brian Behlendorf [Wed, 13 Nov 2019 17:23:14 +0000 (09:23 -0800)]
Fix `zpool create -o <property>` error message
When `zpool create -o <property>` is run without root permissions
and the pool property requested is not specifically enumerated in
zpool_valid_proplist(). Then an incorrect error message referring
to an invalid property is printed rather than the expected permission
denied error.
Specifying a pool property at create time should be handled the same
way as filesystem properties in zfs_valid_proplist(). There should
not be default zfs_error_aux() set for properties which are not
listed.
Ross Williams [Wed, 13 Nov 2019 17:21:07 +0000 (12:21 -0500)]
Reorganize zpool(8) man page into sections
Moved subcommand topics into individual manpages. Reordered and
grouped the list of subcommands by topic.
Moved concepts overview to `zpoolconcepts.8` and the long list of
available pool properties to `zpoolprops.8`.
Internal cross-references copied from `zpool.8` needed to be
converted to `.Xr` external references to new subcommand manual
pages.
Move `autotrim` into lexical order, autotrim tacked onto the end
of a list. Now it is in alphabetical order.
Clarify attach/detach description. Description was too specific to
command syntax. Overview clarifies reason for attaching or detaching
a device.
Clarify replace description, don't refer to subcommand arguments.
Clarify split command description, say what split actually does and
why you'd want to do it.
Clarify description of upgrade, and simplify the zpool.8 wording of
the zpool-upgrade(8) description.
Clarify description of import, detail what zpool-import(8) actually
does.
Add appropriate SEE ALSO sections. Divided zpool subcommand manual
pages need their own SEE ALSO sections. Also modified fsck.zfs.8
to point directly to zfs-scrub.8 and zed.8.in to include a direct
reference to zfs-events.8
Reviewed-by: Matt Ahrens <matt@delphix.com> Reviewed-by: Kjeld Schouten <kjeld@schouten-lebbing.nl> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Ross Williams <ross@ross-williams.net>
Closes #9564
Ross Williams [Tue, 12 Nov 2019 19:17:40 +0000 (14:17 -0500)]
Reorganize zfs(8) man page into sections
Most subcommands got their own manpages (e.g. create). Some related
commands grouped into a single manpage and symlinks created (e.g. set,
get, and inherit). I did this when topics were either too short to
warrant their own file or so interrelated that a user would want to
refer between commands in the same file.
Corrected .Sx internal references to .Xr cross refs; lots of .Sx
references from when text was all in zfs.8 needed to be changed to
.Xr zfs-$SUBCOMMAND 8 cross references.
Divided subcommand list in zfs(8) into sections of related
functionality. This required writing new descriptions for some
commands.
Preserved ".Os Linux", `.Os` macro parsing behavior differs between
mandoc from the "BSD" mandoc package (available on Ubuntu) and man
from Ubuntu's man-db package, which calls groff to format the manpages.
Groff handles the `.Os` macro differently and wrongly, defaulting
it to "BSD" in `/usr/share/groff/*/tmac/mdoc/doc-common`, instead of
getting the default from `uname`.
A future set of changes will introduce build-time preprocessing of
manpages for platform-specific documentation and can insert the
correct operating system name.
Added SEE ALSO sections, the newly-divided zfs-*.8 subcommand man
pages needed their own SEE ALSO sections pointing to related
subcommands and, in some cases, documentation from other packages
(e.g. zfs-share.8).
Reviewed-by: Matt Ahrens <matt@delphix.com> Reviewed-by: Kjeld Schouten <kjeld@schouten-lebbing.nl> Reviewed-by: Sean Eric Fagan <sef@ixsystems.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Ross Williams <ross@ross-williams.net>
Closes #9559
Matthew Macy [Tue, 12 Nov 2019 18:40:39 +0000 (10:40 -0800)]
Add wrapper stub for zfs_cmd ioctl to libzpool
FreeBSD needs a wrapper for handling zfs_cmd ioctls.
In libzfs this is handled by zfs_ioctl. However, here
we need to wrap the call directly.
Reviewed-by: Allan Jude <allanjude@freebsd.org> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9511
Brian Behlendorf [Tue, 12 Nov 2019 16:59:06 +0000 (08:59 -0800)]
Linux compat: Minimum kernel version 3.10
Increase the minimum supported kernel version from 2.6.32 to 3.10.
This removes support for the following Linux enterprise distributions.
Distribution | Kernel | End of Life
---------------- | ------ | -------------
Ubuntu 12.04 LTS | 3.2 | Apr 28, 2017
SLES 11 | 3.0 | Mar 32, 2019
RHEL / CentOS 6 | 2.6.32 | Nov 30, 2020
The following changes were made as part of removing support.
* Updated `configure` to enforce a minimum kernel version as
specified in the META file (Linux-Minimum: 3.10).
configure: error:
*** Cannot build against kernel version 2.6.32.
*** The minimum supported kernel version is 3.10.
* Removed all `configure` kABI checks and matching C code for
interfaces which solely predate the Linux 3.10 kernel.
* Updated all `configure` kABI checks to fail when an interface is
missing which was in the 3.10 kernel up to the latest 5.1 kernel.
Removed the HAVE_* preprocessor defines for these checks and
updated the code to unconditionally use the verified interface.
* Inverted the detection logic in several kABI checks to match
the new interface as it appears in 3.10 and newer and not the
legacy interface.
* Consolidated the following checks in to individual files. Due
the large number of changes in the checks it made sense to handle
this now. It would be desirable to group other related checks in
the same fashion, but this as left as future work.
- config/kernel-blkdev.m4 - Block device kABI checks
- config/kernel-blk-queue.m4 - Block queue kABI checks
- config/kernel-bio.m4 - Bio interface kABI checks
* Removed the kABI checks for sops->nr_cached_objects() and
sops->free_cached_objects(). These interfaces are currently unused.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #9566
Ryan Moeller [Mon, 11 Nov 2019 20:15:44 +0000 (15:15 -0500)]
Allow platform dependent path stripping for vdevs
On Linux the full path preceding devices is stripped when formatting
vdev names. On FreeBSD we only want to strip "/dev/". Hide the
implementation details of path stripping behind zfs_strip_path().
Make zfs_strip_partition_path() static in Linux implementation while
here, since it is never used outside of the file it is defined in.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Ryan Moeller <ryan@ixsystems.com>
Closes #9565
Pavel Snajdr [Mon, 11 Nov 2019 17:34:21 +0000 (18:34 +0100)]
Remove zpl_revalidate
This patch removes the need for zpl_revalidate altogether.
There were 3 main reasons why we used d_revalidate:
1. periodic automounted snapshots umount deferral
2. negative dentries created before snapshot rollback
3. stale inodes referenced by dentry cache after snapshot rollback
Periodic snapshots deferral solution introduces zfs_exit_fs function,
which is called as a part of ZFS_EXIT(zfsvfs_t) macro.
Negative dentries and stale inodes are solved by flushing the dcache
for the particular dataset on zfs_resume_fs call.
This patch also removes now unused HAVE_S_D_OP configure test.
Reviewed-by: Aleksa Sarai <cyphar@cyphar.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Pavel Snajdr <snajpa@snajpa.net>
Closes #8774
Closes #9549
Alexander Motin [Mon, 11 Nov 2019 17:27:59 +0000 (20:27 +0300)]
Improve logging of 128KB writes
Before my ZIL space optimization few years ago 128KB writes were logged
as two 64KB+ records in two 128KB log blocks. After that change it
became ~127KB+/1KB+ in two 128KB log blocks to free space in the second
block for another record. Unfortunately in case of 128KB only writes,
when space in the second block remained unused, that change increased
write latency by unbalancing checksum computation and write times
between parallel threads. It also didn't help with SLOG space
efficiency in that case.
This change introduces new 68KB log block size, used for both writes
below 67KB and 128KB-sharp writes. Writes of 68-127KB are still using
one 128KB block to not increase processing overhead. Writes above
131KB are still using full 128KB blocks, since possible saving there
is small. Mixed loads will likely also fall back to previous 128KB,
since code uses maximum of the last 16 requested block sizes.
Reviewed-by: Matt Ahrens <matt@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Closes #9409
Witaut Bajaryn [Fri, 8 Nov 2019 22:34:07 +0000 (23:34 +0100)]
Skip loading already loaded key
Don't ask for the password / try to load the key if the key for the
encryptionroot is already loaded. The user might have loaded the key
manually or by other means before the scripts get called.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Tom Caputi <tcaputi@datto.com> Reviewed-by: Richard Laager <rlaager@wiktel.com> Signed-off-by: Witaut Bajaryn <vitaut.bayaryn@gmail.com>
Closes #9495
Closes #9529
Pavel Zakharov [Wed, 6 Nov 2019 18:51:19 +0000 (13:51 -0500)]
zvol_wait should ignore redacted zvols
zvol_wait waits for zvol links to be created under /dev/zvol for each zvol.
Links are not created for redacted zvols so we should ignore those.
Reviewed-by: Paul Dagnelie <pcd@delphix.com> Reviewed-by: Matt Ahrens <matt@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Pavel Zakharov <pavel.zakharov@delphix.com>
Closes #9545
Prakash Surya [Thu, 24 Oct 2019 21:13:41 +0000 (14:13 -0700)]
Add tracepoints for taskq entry lifetime events
This adds some new DTRACE_PROBE* endpoints so that we can observe taskq
latencies on a system. Additionally, a new "taskqlatency.bt" script is
added to do this observation via "bpftrace". Lastly, a "zfs-trace.sh"
script is added to wrap "bpftrace" with the proper options required to
run and use "taskqlatency.bt".
For example, with these changes in place, a user can run the following:
Prakash Surya [Wed, 30 Oct 2019 18:02:41 +0000 (11:02 -0700)]
Enable use of DTRACE_PROBE* macros in "spl" module
This change modifies some of the infrastructure for enabling the use of
the DTRACE_PROBE* macros, such that we can use tehm in the "spl" module.
Currently, when the DTRACE_PROBE* macros are used, they get expanded to
create new functions, and these dynamically generated functions become
part of the "zfs" module.
Since the "spl" module does not depend on the "zfs" module, the use of
DTRACE_PROBE* in the "spl" module would result in undefined symbols
being used in the "spl" module. Specifically, DTRACE_PROBE* would turn
into a function call, and the function being called would be a symbol
only contained in the "zfs" module; which results in a linker and/or
runtime error.
Thus, this change adds the necessary logic to the "spl" module, to
mirror the tracing functionality available to the "zfs" module. After
this change, we'll have a "trace_zfs.h" header file which defines the
probes available only to the "zfs" module, and a "trace_spl.h" header
file which defines the probes available only to the "spl" module.
Reviewed by: Brad Lewis <brad.lewis@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Prakash Surya <prakash.surya@delphix.com>
Closes #9525
Matthew Macy [Fri, 1 Nov 2019 17:37:33 +0000 (10:37 -0700)]
Prefix struct rangelock
A struct rangelock already exists on FreeBSD. Add a zfs_ prefix as
per our convention to prevent any conflict with existing symbols.
This change is a follow up to 2cc479d0.
Reviewed-by: Matt Ahrens <matt@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9534
Brian Behlendorf [Thu, 31 Oct 2019 19:57:42 +0000 (12:57 -0700)]
ZTS: Fix removal_with_errors
The removal_with_errors.ksh test case could occasionally complete
the removal process instead of canceling due to an injected error.
To prevent this false positive, export and import the pool between
test phases to flush the ARC cache. Furthermore, double the amount
of data in the pool to increase the removal time.
Reviewed-by: Ryan Moeller <ryan@ixsystems.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #9528
Reviewed-by: Matt Ahrens <matt@delphix.com> Reviewed-by: Igor Kozhukhov <igor@dilos.org> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9533
Matthew Macy [Thu, 31 Oct 2019 17:09:01 +0000 (10:09 -0700)]
Include prototypes for vdev_initialize
Address two prototype related warnings emitted by clang.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Igor Kozhukhov <igor@dilos.org> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9535
Matthew Macy [Thu, 31 Oct 2019 16:52:22 +0000 (09:52 -0700)]
Factor Linux specific code out of spa_misc.c
Move these Linux module parameter get/set helpers in to
platform specific code.
Reviewed-by: Igor Kozhukhov <igor@dilos.org> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Ryan Moeller <ryan@ixsystems.com> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9457
alaviss [Wed, 30 Oct 2019 21:38:41 +0000 (21:38 +0000)]
dracut/zfs-load-key.sh: properly remove prefixes
Removes the 'ZFS=' prefix from $BOOTFS instead of $root. This makes sure
that the 'zfs:' prefix remains stripped so that users with
'root=zfs:dataset' cmdline can have key loaded on boot again.
Reviewed-by: Garrett Fields <ghfields@gmail.com> Reviewed-by: Dacian Reece-Stremtan <dacianstremtan@gmail.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Hiếu Lê <leorize+oss@disroot.org>
Closes #9520
Tom Caputi [Wed, 30 Oct 2019 18:27:28 +0000 (14:27 -0400)]
Fix 'zfs change-key' with unencrypted child
Currently, when you call 'zfs change-key' on an encrypted dataset
that has an unencrypted child, the code will trigger a VERIFY.
This VERIFY is leftover from before we allowed unencrypted
datasets to exist underneath encrypted ones. This patch fixes the
issue by simply replacing the VERIFY with an early return when
recursing through datasets.
Reviewed by: Jason King <jason.brian.king@gmail.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Igor Kozhukhov <igor@dilos.org> Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #9524
Matthew Macy [Mon, 28 Oct 2019 16:51:53 +0000 (09:51 -0700)]
Minor spa portability fixes
- FreeBSD's rootpool import code uses spa_config_parse
- Move the zvol_create_minors call out from under the
spa_namespace_lock in spa_import. It isn't needed and it causes
a lock order reversal on FreeBSD.
Reviewed-by: Igor Kozhukhov <igor@dilos.org> Reviewed-by: Jorgen Lundman <lundman@lundman.net> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9499
Chunwei Chen [Mon, 28 Oct 2019 16:49:44 +0000 (09:49 -0700)]
Fix zpool history unbounded memory usage
In original implementation, zpool history will read the whole history
before printing anything, causing memory usage goes unbounded. We fix
this by breaking it into read-print iterations.
Reviewed-by: Tom Caputi <tcaputi@datto.com> Reviewed-by: Matt Ahrens <matt@delphix.com> Reviewed-by: Igor Kozhukhov <igor@dilos.org> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Chunwei Chen <david.chen@nutanix.com>
Closes #9516
loli10K [Sat, 26 Oct 2019 22:22:19 +0000 (00:22 +0200)]
Fix for ARC sysctls ignored at runtime
This change leverage module_param_call() to run arc_tuning_update()
immediately after the ARC tunable has been updated as suggested in cffa8372 code review.
A simple test case is added to the ZFS Test Suite to prevent future
regressions in functionality.
Reviewed-by: Matt Macy <mmacy@FreeBSD.org> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #9487
Closes #9489
Matthew Macy [Sat, 26 Oct 2019 17:07:59 +0000 (10:07 -0700)]
Remove unneeded header from libzpool/kernel.c
The sys/signal.h header doesn't exist on FreeBSD, nor is
it needed on Linux.
Reviewed-by: Igor Kozhukhov <igor@dilos.org> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9510
Ryan Moeller [Sat, 26 Oct 2019 03:03:46 +0000 (23:03 -0400)]
ZTS: Move more tests to linux.run
Tests that rely on special filesystems that are specific to Linux
should only be run on Linux.
Reviewed-by: Igor Kozhukhov <igor@dilos.org> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Giuseppe Di Natale <guss80@gmail.com> Signed-off-by: Ryan Moeller <ryan@ixsystems.com>
Closes #9512
Tom Caputi [Thu, 24 Oct 2019 17:51:01 +0000 (13:51 -0400)]
Fix incremental recursive encrypted receive
Currently, incremental recursive encrypted receives fail to work
for any snapshot after the first. The reason for this is because
the check in zfs_setup_cmdline_props() did not properly realize
that when the user attempts to use '-x encryption' in this
situation, they are not really overriding the existing encryption
property and instead are attempting to prevent it from changing.
This resulted in an error message stating: "encryption property
'encryption' cannot be set or excluded for raw or incremental
streams".
This problem is fixed by updating the logic to expect this use
case.
Reviewed-by: loli10K <ezomori.nozomu@gmail.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Igor Kozhukhov <igor@dilos.org> Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #9494
Brian Behlendorf [Thu, 24 Oct 2019 17:17:33 +0000 (10:17 -0700)]
Linux 4.14, 4.19, 5.0+ compat: SIMD save/restore
Contrary to initial testing we cannot rely on these kernels to
invalidate the per-cpu FPU state and restore the FPU registers.
Nor can we guarantee that the kernel won't modify the FPU state
which we saved in the task struck.
Therefore, the kfpu_begin() and kfpu_end() functions have been
updated to save and restore the FPU state using our own dedicated
per-cpu FPU state variables.
This has the additional advantage of allowing us to use the FPU
again in user threads. So we remove the code which was added to
use task queues to ensure some functions ran in kernel threads.
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com> Reviewed-by: Tony Hutter <hutter2@llnl.gov> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #9346
Closes #9403
chrisrd [Wed, 23 Oct 2019 22:26:17 +0000 (09:26 +1100)]
Don't call arc_buf_destroy on unallocated arc_buf
Fixes an obvious issue of calling arc_buf_destroy() on an
unallocated arc_buf.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Matt Ahrens <matt@delphix.com> Reviewed-by: Paul Dagnelie <pcd@delphix.com> Signed-off-by: Chris Dunlop <chris@onthe.net.au>
Closes #9453
Matthew Macy [Wed, 23 Oct 2019 20:48:31 +0000 (13:48 -0700)]
Use platform independent error code for libzfs_run_process_impl
Reviewed-by: Tony Hutter <hutter2@llnl.gov> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9493
Matthew Macy [Mon, 21 Oct 2019 03:37:30 +0000 (20:37 -0700)]
Use correct format string when printing int8
Reviewed-by: Igor Kozhukhov <igor@dilos.org> Reviewed-by: Ryan Moeller <ryan@ixsystems.com> Reviewed-by: George Melikov <mail@gmelikov.ru> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9486
Matthew Macy [Sun, 20 Oct 2019 00:08:19 +0000 (17:08 -0700)]
Remove gratuitous Linux only include in ztest & zdb
We don't need to include stdio_ext.h
Reviewed-by: Igor Kozhukhov <igor@dilos.org> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9483
With 4k disks, this test will fail in the last section because the
expected human readable value of 20.0M is reported as 20.1M. Rather than
use the human readable property, switch to the parsable property and
verify that the values are reasonably close.
Reviewed-by: Igor Kozhukhov <igor@dilos.org> Reviewed-by: Ryan Moeller <ryan@ixsystems.com> Reviewed-by: Tony Hutter <hutter2@llnl.gov> Signed-off-by: John Kennedy <john.kennedy@delphix.com>
Closes #9477
Giving a name to this enum makes it discoverable from
debugging tools like DRGN and SDB. For example, with
the name proposed on this patch we can iterate over
these values in DRGN:
```
>>> prog.type('enum kmc_bit').enumerators
(('KMC_BIT_NOTOUCH', 0), ('KMC_BIT_NODEBUG', 1),
('KMC_BIT_NOMAGAZINE', 2), ('KMC_BIT_NOHASH', 3),
('KMC_BIT_QCACHE', 4), ('KMC_BIT_KMEM', 5),
('KMC_BIT_VMEM', 6), ('KMC_BIT_SLAB', 7),
...
```
This enables SDB to easily pretty-print the flags of
the spl_kmem_caches in the system like this:
```
> spl_kmem_caches -o "name,flags,total_memory"
name flags total_memory
------------------------ ----------------------- ------------
abd_t KMC_NOMAGAZINE|KMC_SLAB 4.5MB
arc_buf_hdr_t_full KMC_NOMAGAZINE|KMC_SLAB 12.3MB
... <cropped> ...
ddt_cache KMC_VMEM 583.7KB
ddt_entry_cache KMC_NOMAGAZINE|KMC_SLAB 0.0B
... <cropped> ...
zio_buf_1048576 KMC_NODEBUG|KMC_VMEM 0.0B
... <cropped> ...
```
Reviewed-by: Matt Ahrens <matt@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #9478
Update skc_obj_alloc for spl kmem caches that are backed by Linux
Currently, for certain sizes and classes of allocations we use
SPL caches that are backed by caches in the Linux Slab allocator
to reduce fragmentation and increase utilization of memory. The
way things are implemented for these caches as of now though is
that we don't keep any statistics of the allocations that we
make from these caches.
This patch enables the tracking of allocated objects in those
SPL caches by making the trade-off of grabbing the cache lock
at every object allocation and free to update the respective
counter.
Additionally, this patch makes those caches visible in the
/proc/spl/kmem/slab special file.
As a side note, enabling the specific counter for those caches
enables SDB to create a more user-friendly interface than
/proc/spl/kmem/slab that can also cross-reference data from
slabinfo. Here is for example the output of one of those
caches in SDB that outputs the name of the underlying Linux
cache, the memory of SPL objects allocated in that cache,
and the percentage of those objects compared to all the
objects in it:
```
> spl_kmem_caches | filter obj.skc_name == "zio_buf_512" | pp
name ... source total_memory util
----------- ... ----------------- ------------ ----
zio_buf_512 ... kmalloc-512[SLUB] 16.9MB 8
```
Reviewed-by: Matt Ahrens <matt@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #9474
Ryan Moeller [Thu, 17 Oct 2019 02:19:48 +0000 (22:19 -0400)]
Detect if sed supports --in-place
Not all versions of sed have the --in-place flag. Detect support for
the flag during ./configure and provide a fallback mechanism for those
systems where sed's behavior differs. The autoconf variable
${ac_inplace} can be used to choose the correct flags for editing a
file in place with sed.
Replace violating usages in Makefile.am with ${ac_inplace}.
Reviewed-by: Chris Dunlop <chris@onthe.net.au> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Ryan Moeller <ryan@ixsystems.com>
Closes #9463
Matthew Macy [Thu, 17 Oct 2019 01:43:52 +0000 (18:43 -0700)]
Make zfsdev_getminor signature cross platform
Only pass the file descriptor to make zfsdev_get_miror() portable.
Reviewed-by: George Melikov <mail@gmelikov.ru> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9466
Matthew Macy [Thu, 17 Oct 2019 01:37:31 +0000 (18:37 -0700)]
Move linux specific mmp module_param_call handler to platform code
Reviewed-by: Ryan Moeller <ryan@ixsystems.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9465
Matthew Macy [Thu, 17 Oct 2019 01:36:16 +0000 (18:36 -0700)]
Add default case to lua kernel code
Some platforms, e.g. FreeBSD, support user space setjmp
semantics in kernel.
Reviewed-by: Igor Kozhukhov <igor@dilos.org> Reviewed-by: Ryan Moeller <ryan@ixsystems.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9450
Matthew Macy [Thu, 17 Oct 2019 01:34:19 +0000 (18:34 -0700)]
Fix signature for private functions without header declarations
Clang will complain if a function has no prior declaration
Reviewed-by: Ryan Moeller <ryan@ixsystems.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9467
Matthew Macy [Mon, 14 Oct 2019 02:25:19 +0000 (19:25 -0700)]
Remove dead code and cleanup scoping in dmu_send.c
This addresses a number of problems with dmu_send.c:
* bp_span is unused which makes clang complain
* dump_write conflicts with FreeBSD's existing core dump code
* range_alloc is private to the file and not declared in any headers
causing clang to complain
Reviewed-by: Matt Ahrens <matt@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9432
Paul Dagnelie [Mon, 14 Oct 2019 02:21:51 +0000 (19:21 -0700)]
Don't call sizeof on void
We get the sizeof the appropriate type, and don't cast away const.
Reviewed-by: Ryan Moeller <ryan@ixsystems.com> Reviewed-by: Igor Kozhukhov <igor@dilos.org> Reviewed-by: Matt Macy <mmacy@FreeBSD.org> Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #9455
Matthew Macy [Mon, 14 Oct 2019 02:19:39 +0000 (19:19 -0700)]
Move zfs_onexit_fd_hold to platform code
FreeBSD has a very different implementation.
Reviewed-by: Ryan Moeller <ryan@ixsystems.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9442
Matthew Macy [Mon, 14 Oct 2019 02:15:27 +0000 (19:15 -0700)]
Move zio_delay_interrupt to platform code
FreeBSD has its own implementation as do other platforms.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Ryan Moeller <ryan@ixsystems.com> Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #9439
Brian Behlendorf [Mon, 14 Oct 2019 02:13:26 +0000 (19:13 -0700)]
Modify sharenfs=on default behavior
While it may sometimes be convenient to export an NFS filesystem with
no_root_squash it should not be the default behavior. Align the
default behavior with the Linux NFS server defaults. To restore
the previous behavior use 'zfs set sharenfs="no_root_squash,..."'.
Reviewed-by: loli10K <ezomori.nozomu@gmail.com> Reviewed-by: Richard Laager <rlaager@wiktel.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #9397
Closes #9425
chrisrd [Fri, 11 Oct 2019 17:33:13 +0000 (04:33 +1100)]
Typo fix in comment: dso_dryrun
Reviewed-by: Ryan Moeller <ryan@ixsystems.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Chris Dunlop <chris@onthe.net.au>
Closes #9452
Brian Behlendorf [Fri, 11 Oct 2019 17:22:20 +0000 (10:22 -0700)]
ZTS: Fix zpool_status_-s
After commit 5e74ac51 which split and reordered the run files the
`zpool_status_-s` test began failing. The new ordering placed
the test after a previous test which used `zpool replace` to replace
a disk but did not clear its label. This resulted in the next test,
`zpool_status_-s`, failing because of the potentially active
pool being detected on the replaced vdev.
/dev/loop0 is part of potentially active pool 'testpool'
Use the default_mirror_setup_noexit() and default_cleanup_noexit()
functions to create the pool in `zpool_status_-s`. They use the -f
flag by default.
In the `scrub_after_resilver` test wipe the label during cleanup
to prevent future failures if the tests are again reordered.
Reviewed-by: Igor Kozhukhov <igor@dilos.org> Reviewed-by: Ryan Moeller <ryan@ixsystems.com> Reviewed-by: John Kennedy <john.kennedy@delphix.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #9451
Paul Dagnelie [Fri, 11 Oct 2019 17:13:21 +0000 (10:13 -0700)]
Function name and comment updates
Rename certain functions for more consistency when they share common
features. Make comments clearer about what arguments should be passed
to the insert and add functions.
Reviewed by: Sara Hartse <sara.hartse@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Matt Ahrens <matt@delphix.com> Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #9441