Ryan Moeller [Mon, 26 Jul 2021 20:08:52 +0000 (16:08 -0400)]
ZTS: Add tests for creation time
Reviewed-by: Tony Nguyen <tony.nguyen@delphix.com> Reviewed-by: Allan Jude <allan@klarasystems.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Alexander Motin <mav@FreeBSD.org> Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #12432
Richard Yao [Sun, 17 Mar 2019 00:43:13 +0000 (20:43 -0400)]
Linux 4.11 compat: statx support
Linux 4.11 added a new statx system call that allows us to expose crtime
as btime. We do this by caching crtime in the znode to match how atime,
ctime and mtime are cached in the inode.
statx also introduced a new way of reporting whether the immutable,
append and nodump bits have been set. It adds support for reporting
compression and encryption, but the semantics on other filesystems is
not just to report compression/encryption, but to allow it to be turned
on/off at the file level. We do not support that.
We could implement semantics where we refuse to allow user modification
of the bit, but we would need to do a dnode_hold() in zfs_znode_alloc()
to find out encryption/compression information. That would introduce
locking that will have a minor (although unmeasured) performance cost.
It also would be inferior to zdb, which reports far more detailed
information. We therefore omit reporting of encryption/compression
through statx in favor of recommending that users interested in such
information use zdb.
Reviewed-by: Tony Nguyen <tony.nguyen@delphix.com> Reviewed-by: Allan Jude <allan@klarasystems.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Alexander Motin <mav@FreeBSD.org> Reviewed-by: Ryan Moeller <ryan@iXsystems.com> Signed-off-by: Richard Yao <ryao@gentoo.org>
Closes #8507
The comment only specifies MNT_ROOTFS - which is set by the kernel when
mounting its root file system. So it's not clear if any other flags
are not quite right and for what reason.
Gordon Bergling [Tue, 17 Aug 2021 17:01:07 +0000 (19:01 +0200)]
zfs.4: Fix typo s/compatiblity/compatibility/
Reviewed-by: George Melikov <mail@gmelikov.ru> Reviewed-by: Ryan Moeller <ryan@ixsystems.com> Reviewed-by: Tony Nguyen <tony.nguyen@delphix.com> Signed-off-by: Gordon Bergling <gbergling@googlemail.com>
Closes #12464
Alex Richardson [Tue, 17 Aug 2021 16:44:40 +0000 (17:44 +0100)]
Mark LLDB/CLANG_BOOTSTRAP/LLD_BOOTSTRAP as broken on non-FreeBSD for now
I enabled these options again in 31ba4ce8898f9dfa5e7f054fdbc26e50a599a6e3,
but unfortunately only my specific build configuration worked whereas the
build with default options is still broken.
Alexander Motin [Tue, 17 Aug 2021 16:15:54 +0000 (12:15 -0400)]
Remove b_pabd/b_rabd allocation from arc_hdr_alloc()
When a header is allocated for full overwrite it is a waste of time
to allocate b_pabd/b_rabd for it, since arc_write() will free them
without ever being touched. If it is a read or a partial overwrite
then arc_read() and arc_hdr_decrypt() allocate them explicitly.
Reduced memory allocation in user threads also reduces ARC eviction
throttling there, proportionally increasing it in ZIO threads, that
is not good. To minimize or even avoid it introduce ARC allocation
reserve, allowing certain arc_get_data_abd() callers to allocate a
bit longer in situations where user threads will already throttle.
Reviewed-by: George Wilson <gwilson@delphix.com> Reviewed-by: Mark Maybee <mark.maybee@delphix.com> Signed-off-by: Alexander Motin <mav@FreeBSD.org> Sponsored-By: iXsystems, Inc.
Closes #12398
Ed Maste [Tue, 17 Aug 2021 15:58:03 +0000 (11:58 -0400)]
sysctl.9: put negative sense sysctl note in own paragraph
The sysctl man page cautions against negative-sense boolean sysctls
(foobar_disable), but it gets lost at the end of a large paragraph.
Move it to a separate paragraph in an attempt to make it more clear.
This man page could use a more holistic review and edit pass. This
change is simple and straightforward and I hope provides a small but
immediate benefit.
Alexander Motin [Tue, 17 Aug 2021 15:59:46 +0000 (11:59 -0400)]
Increase default volblocksize from 8KB to 16KB
Many things has changed since previous default was set many years ago.
Nowadays 8KB does not allow adequate compression or even decent space
efficiency on many of pools due to 4KB disk physical block rounding,
especially on RAIDZ and DRAID. It effectively limits write throughput
to only 2-3GB/s (250-350K blocks/s) due to sync thread, allocation,
vdev queue and other block rate bottlenecks. It keeps L2ARC expensive
despite many optimizations and dedup just unrealistic.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: George Melikov <mail@gmelikov.ru> Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Closes #12406
Alexander Motin [Tue, 17 Aug 2021 15:55:34 +0000 (11:55 -0400)]
Optimize arc_l2c_only lists assertions
It is very expensive and not informative to call multilist_is_empty()
for each arc_change_state() on debug builds to check for impossible.
Instead implement special index function for arc_l2c_only->arcs_list,
multilists, panicking on any attempt to use it.
Reviewed-by: Mark Maybee <mark.maybee@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Alexander Motin <mav@FreeBSD.org> Sponsored-By: iXsystems, Inc.
Closes #12421
Alexander Motin [Tue, 17 Aug 2021 15:50:31 +0000 (11:50 -0400)]
Fix/improve dbuf hits accounting
Instead of clearing stats inside arc_buf_alloc_impl() do it inside
arc_hdr_alloc() and arc_release(). It fixes statistics being wiped
every time a new dbuf is filled from the ARC.
Remove b_l1hdr.b_l2_hits. L2ARC hits are accounted at b_l2hdr.b_hits.
Since the hits are accounted under hash lock, replace atomics with
simple increments.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: George Wilson <george.wilson@delphix.com> Signed-off-by: Alexander Motin <mav@FreeBSD.org> Sponsored-By: iXsystems, Inc.
Closes #12422
Alexander Motin [Tue, 17 Aug 2021 15:47:00 +0000 (11:47 -0400)]
Avoid vq_lock drop in vdev_queue_aggregate()
vq_lock is already too congested for two more operations per I/O.
Instead of dropping and reacquiring it inside vdev_queue_aggregate()
delegate the zio_vdev_io_bypass() and zio_execute() calls for parent
I/Os to callers, that drop the lock any way to execute the new I/O.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Mark Maybee <mark.maybee@delphix.com> Reviewed-by: Brian Atkinson <batkinson@lanl.gov> Signed-off-by: Alexander Motin <mav@FreeBSD.org> Sponsored-By: iXsystems, Inc.
Closes #12297
Alexander Motin [Tue, 17 Aug 2021 15:44:34 +0000 (11:44 -0400)]
Use more atomics in refcounts
Use atomic_load_64() for zfs_refcount_count() to prevent torn reads
on 32-bit platforms. On 64-bit ones it should not change anything.
When built with ZFS_DEBUG but running without tracking enabled use
atomics instead of mutexes same as for builds without ZFS_DEBUG.
Since rc_tracked can't change live we can check it without lock.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Matthew Ahrens <mahrens@delphix.com> Signed-off-by: Alexander Motin <mav@FreeBSD.org> Sponsored-By: iXsystems, Inc.
Closes #12420
rtld: avoid use of of getenv(3) for evaluating rtld env vars (LD_XXX)
Scan through the set of environment variables during initialization and
store values in the corresponding ld_env_var_desc structure, in the
single pass at init time. This does not eliminate use of getenv(3) and
unsetenv(3) completely, but provides a foundation to do that as the next
step.
Also organize the scan in a way that makes it easier to support aliases
like LD_DEBUG vs. LD_64_DEBUG.
Suggested by: arichardson
Reviewed by: arichardson, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D31545
Instead of specifying the main name part of the environment variable as the
string literal, create array of the var names and access them by symbolic
index. Convert main name parts into complete names by prefixing with
ABI-specific ld_env_vars.
This way the name is not repeated, and also it can carry additional
proporties explicitly. For instance, cleanup of the environment for
the setuid image does not require retyping all names.
Reviewed by: arichardson, markj
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D31545
Dan Langille [Sun, 15 Aug 2021 16:53:16 +0000 (12:53 -0400)]
Enable rc.d/jail within jails
Jails with jails is a supported. This change allows the script to run
upon startup with a jail. Without this, jails are not automatically
started within jails.
ipfw: fix possible data race between jump cache reading and updating.
Jump cache is used to reduce the cost of rule lookup for O_SKIPTO and
O_CALLRETURN actions. It uses rules chain id to check correctness of
cached value. But due to the possible race, there is the chance that
one thread can read invalid value. In some cases this can lead to out
of bounds access and panic.
Use thread fence operations to constrain the reordering of accesses.
Also rename jump_fast and jump_linear functions to jump_cached and
jump_lookup_pos respectively.
Ryan Moeller [Mon, 16 Aug 2021 23:38:34 +0000 (19:38 -0400)]
ZTS: Avoid unset $tmpdir in redacted_panic
The redacted_send tests make use of a $tmpdir variable, except in
redacted_send/redacted_panic the variable is never defined.
Use $TEST_BASE_DIR instead.
Clean up the stream file after the test.
Reviewed-by: John Kennedy <john.kennedy@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #12455
Allow consistency validation of the inet6 fib based on rib data.
Validation can be kicked off by loading test_lookup module and
running sysctl net.route.test.run_inet6_scan=1
The macro bit_foreach() traverses all set bits in the bitstring in the
forward direction, assigning each location in turn to variable.
The macro bit_foreach_at() traverses all set bits in the bitstring in
the forward direction at or after the zero-based bit index, assigning
each location in turn to variable.
The bit_foreach_unset() and bit_foreach_unset_at() macros which
traverses unset bits are implemented for completeness.
Mark Johnston [Mon, 16 Aug 2021 17:15:25 +0000 (13:15 -0400)]
sigtimedwait: Use a unique wait channel for sleeping
When a sigtimedwait(2) caller goes to sleep, it uses a wait channel of
p->p_sigacts with the proc lock as the interlock. However, p_sigacts
can be shared between processes if a child is created with
rfork(RFSIGSHARE | RFPROC). Thus we can end up with two threads
sleeping on the same wait channel using different locks, which is not
permitted.
Fix the problem simply by using a process-unique wait channel, following
the example of sigsuspend. The actual wait channel value is irrelevant
here, sleeping threads are awoken using sleepq_abort().
John Baldwin [Mon, 16 Aug 2021 17:42:46 +0000 (10:42 -0700)]
ktls: Fix accounting for TLS 1.0 empty fragments.
TLS 1.0 empty fragment mbufs have no payload and thus m_epg_npgs is
zero. However, these mbufs need to occupy a "unit" of space for the
purposes of M_NOTREADY tracking similar to regular mbufs. Previously
this was done for the page count returned from ktls_frame() and passed
to ktls_enqueue() as well as the page count passed to pru_ready().
However, sbready() and mb_free_notready() only use m_epg_nrdy to
determine the number of "units" of space in an M_EXT mbuf, so when a
TLS 1.0 fragment was marked ready it would mark one unit of the next
mbuf in the socket buffer as ready as well. To fix, set m_epg_nrdy to
1 for empty fragments. This actually simplifies the code as now only
ktls_frame() has to handle TLS 1.0 fragments explicitly and the rest
of the KTLS functions can just use m_epg_nrdy.
Dimitry Andric [Mon, 16 Aug 2021 16:56:41 +0000 (18:56 +0200)]
Apply upstream lldb fix for unhandled Error causing abort
Merge commit 5033f0793fe6 from llvm git (by Dimitry Andric):
[lldb] Avoid unhandled Error in TypeSystemMap::GetTypeSystemForLanguage
When assertions are turned off, the `llvm::Error` value created at the
start of this function is overwritten using the move-assignment
operator, but the success value is never checked. Whenever a TypeSystem
cannot be found or created, this can lead to lldb core dumping with:
Program aborted due to an unhandled Error:
Error value was Success. (Note: Success values must still be checked prior to being destroyed).
Fix this by not creating a `llvm::Error` value in advance, and directly
returning the result of `llvm::make_error` instead, whenever an error is
encountered.
See also: <https://bugs.freebsd.org/253881> and
<https://bugs.freebsd.org/257829>.
Allan Jude [Mon, 16 Aug 2021 15:35:19 +0000 (11:35 -0400)]
Restore FreeBSD sysctl processing for arc.min and arc.max
Before OpenZFS 2.0, trying to set the FreeBSD sysctl vfs.zfs.arc_max
to a disallowed value would return an error.
Since the switch, it instead only generates WARN_IF_TUNING_IGNORED
Keep the ability to set the sysctl's specifically to 0, even though
that is less than the minimum, because some tests depend on this.
Also lost, was the ability to set vfs.zfs.arc_max to a value less
than the default vfs.zfs.arc_min at boot time. Restore this as well.
Reviewed-by: Tony Nguyen <tony.nguyen@delphix.com> Reviewed-by: Ryan Moeller <ryan@ixsystems.com> Signed-off-by: Allan Jude <allan@klarasystems.com>
Closes #12161
Ed Maste [Thu, 25 Jun 2020 00:42:10 +0000 (20:42 -0400)]
bxe: tag files to skip clang-format formatting
bxe contains three files which are sets of constants or other data, and
might be auto-generated or have an upstream. They are rather large
files and clang-format takes quite some time when run against them, so
just skip formatting.
Fangrui Song [Sun, 15 Aug 2021 04:13:33 +0000 (07:13 +0300)]
rtld: Switch to the standard symbol lookup behavior if LD_DYNAMIC_WEAK is set
The current lookup prefers a strong definition to a STB_WEAK definition
(similar to glibc pre-2.2 behavior) which does not conform to the ELF
specification.
The non-compliant behavior provoked https://reviews.llvm.org/D4418
which was intended to fix -shared-libasan but introduced
new problems (and caused some sanitizer tests (e.g.
test/asan/TestCases/interception_failure_test.cpp) to fail): sanitizer
interceptors are STB_GLOBAL instead of STB_WEAK, so defining a second
STB_GLOBAL interceptor can lead to a multiple definition linker error.
For example, in a -fsanitize={address,memory,...} build, libc functions
like malloc/free/strtol/... cannot be provided by user object files.
See
https://docs.freebsd.org/cgi/getmsg.cgi?fetch=16483939+0+archive/2014/freebsd-current/20140716.freebsd-current
for discussions.
This patch implements the ELF-compliant behavior when LD_DYNAMIC_WEAK is
set. STB_WEAK wrestling in symbol lookups in `Search the dynamic linker
itself` are untouched.
For a Variant II architecture, the TP offset of a TLS symbol is st_value -
tlsoffset + r_addend. tlsoffset is computed by either calculate_tls_offset
or calculate_first_tls_offset.
The return value of calculate_first_tls_offset is the smallest integer
satisfying res >= size and (-res) % p_align = p_vaddr % p_align
(= p_offset % p_align). (The formula is a bit contrived. The basic idea
is to subtract the minimum integer from size + align - 1 so that the result
ihas the expected remainder.)
Kyle Evans [Fri, 26 Jun 2020 02:38:27 +0000 (21:38 -0500)]
domain: make it safer to add domains post-domainfinalize
I can see two concerns for adding domains after domainfinalize:
1.) The slow/fast callouts have already been setup.
2.) Userland could create a socket while we're in the middle of
initialization.
We can address #1 fairly easily by tracking whether the domain's been
initialized for at least the default vnet. There are still some concerns
about the callbacks being invoked while a vnet is in the process of
being created/destroyed, but this is a pre-existing issue that the
callbacks must coordinate anyways.
We should also address #2, but technically this has been an issue
anyways because we don't assert on post-domainfinalize additions; we
don't seem to hit it in practice.
Future work can fix that up to make sure we don't find partially
constructed domains, but care must be taken to make sure that at least,
e.g., the usages of pffindproto in ip_input.c can still find them.
Kyle Evans [Fri, 26 Jun 2020 02:17:47 +0000 (21:17 -0500)]
domain: give domains a chance to probe for availability
This gives any given domain a chance to indicate that it's not actually
supported on the current system. If dom_probe isn't supplied, we assume
the domain is universally applicable as most of them are. Keeping
fully-initialized and registered domains around that physically can't
work on a large majority of FreeBSD deployments is sub-optimal and leads
to errors that aren't consistent with the reality of why the socket
can't be created (e.g. ESOCKTNOSUPPORT) because such scenario has to be
caught upon pru_attach, at which point kicking back the more-appropriate
EAFNOSUPPORT would seem weird.
The initial consumer of this will be hvsock, which is only available on
HyperV guests.
Reviewed by: cem (earlier version), bcr (manpages)
Differential Revision: https://reviews.freebsd.org/D25062
Kornel Duleba [Mon, 16 Aug 2021 04:26:50 +0000 (06:26 +0200)]
tpm_tis: Improve interrupt allocation
Validate the irq received from ACPI. Test if it works by sending a simple command and checking if the interrupt handler was executed.
Internal buffer allocation was moved away from common code to tis and crb parts - in order to test the interrupt we need to have it allocated early.
Adam Fenn [Sat, 7 Aug 2021 20:01:46 +0000 (13:01 -0700)]
pvclock: Add 'struct pvclock' API
Consolidate more hypervisor-agnostic functionality behind a new 'struct
pvclock' API.
This should also make it easier to subsequently add hypervisor-agnostic
vDSO timekeeping support.
Also, perform some clean-up:
- Remove 'pvclock_get_last_cycles()'; do not allow external access
to 'pvclock_last_systime' since this is not necessary.
- Consolidate/simplify wall and system time reading codepaths.
- Ensure correct ordering within wall and system time reading
codepaths via 'atomic(9)' and 'rdtsc_ordered()' rather than via
'rmb()'.
- Remove some extra newlines.
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D31418
fstatat(2): handle non-vnode file descriptors for AT_EMPTY_PATH
Set NIRES_EMPTYPATH earlies, to have use of EMPTYPATH recorded even if
we are going to return error. When namei_setup() refused to accept dirfd,
which is not of the vnode type, and indicated by ENOTDIR error return,
fall back to kern_fstat(dirfd).
Reported by: dchagin
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D31530
Ryan Moeller [Fri, 13 Aug 2021 20:42:45 +0000 (16:42 -0400)]
zfs: add missed dependency of zfs module on zlib
Reviewed-by: Alexander Motin <mav@FreeBSD.org> Reviewed-by: Martin Matuska <mm@FreeBSD.org> Co-authored-by: Konstantin Belousov <kib@FreeBSD.org> Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
External-issue: https://reviews.freebsd.org/D31207
Closes #12442
Ryan Moeller [Fri, 13 Aug 2021 20:37:46 +0000 (16:37 -0400)]
Add zfs.sh -r flag to reload modules
zfs.sh already can load and unload, so why not both?
This is convenient when developing changes to the module and you want
to rapidly make some changes, rebuild the module, reload the module,
and test the changes.
Reviewed-by: George Melikov <mail@gmelikov.ru> Reviewed-by: Tony Hutter <hutter2@llnl.gov> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #12450
Ryan Moeller [Fri, 13 Aug 2021 20:13:57 +0000 (16:13 -0400)]
Fix usage of find in tests/Makefile.am
The path is not optional on FreeBSD.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Tony Nguyen <tony.nguyen@delphix.com> Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #12453
ufs rename: ensure that the result of ufs_checkpath() is stable
ufs_rename() calls ufs_checkpath() to ensure that the target directory
is not a child of the source. If not, rename would create a loop.
For instance:
source->X1->X2->target
and if source moved under target, we get corrupted filesystem.
Suppose that we initially have
source->X1 .... and X2->target
where X1 is not on path from root to X2. Then ufs_checkpath() accepts
the inodes, but there is nothing preventing parallel rename of X2 to become
under X1, after checkpath finished.
Ensure stability of ufs_checkpath() result by taking a per-mount sx in
ufs_rename right before ufs_checkpath() and till the end.
Reviewed by: chs, mckusick
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
projects / FreeBSD / FreeBSD.git / log