Stefan Eßer [Sat, 19 Jun 2021 11:27:16 +0000 (13:27 +0200)]
sbin/md5: improve compatibility with coreutils -c option
The -c option expects a digest file in either BSD or coreutils format.
The output for matched and mismatched files is identical to that
of the coreutils version.
The review of these changes included test cases that have already
been committed for the functionality that existed before.
Another test script is added to cover the coreutils compatible
extension implemented by this patch.
This commit contains a tests/Makefile that has been cleaned up
compared to the review version, using an implicit rule to apply the
TESTBASE path at build time (and the scripts have been renamed to
have an extension of .SH instead of .sh to trigger this rule).
Warner Losh [Thu, 20 May 2021 17:26:46 +0000 (11:26 -0600)]
md5: portability fix -- include stdbool.h explicitly
stdbool.h needs to be included to use type bool variables. Due to
namespace pollution, this gets brought in on FreeBSD, but not on
other systems. Include it explicilty.
Warner Losh [Wed, 19 May 2021 17:26:20 +0000 (11:26 -0600)]
md5: Create md5sum, etc compatible programs
On Linux, there's a similar set of programs to ours, but that end in the
letters 'sum'. These act basically like FreeBSD versions run with the -r
option. Add code so that when the program ends in 'sum' you get the
linux -r behavior. This is enough to make most things that use sha*sum
work correctly (the -c / --check options, as well as the long args are
not implemented). When running with the -sum programs, ignore -t instead
of running internal speed tests and make -c an error.
Reviewed by: sef, and kp and allanjude (earlier version)
Relnotes: yes
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D30309
bhyve: implement rdmsr for MSR_IA32_FEATURE_CONTROL
Without the -w option, Windows guests crash on boot. This is caused by a rdmsr
of MSR_IA32_FEATURE_CONTROL. Windows checks this MSR to determine enabled VMX
features. This MSR isn't emulated in bhyve, so a #GP exception is injected
which causes Windows to crash.
Fix by returning a rdmsr of MSR_IA32_FEATURE_CONTROL with Lock Bit set and
VMX disabled to informWindows that VMX isn't available.
Stefan Eßer [Fri, 25 Jun 2021 06:51:40 +0000 (08:51 +0200)]
sbin/md5: add tests
While the correctness of the supported hash algorithms can be tested
with the built-in self-test feature, these test cases are meant to
detect changes in the output format.
A follow-up commit will improve the compatibility with the coreutils
versions of the hash programs, and these tests should detect any
unintended side-effects of such a change on existing functionality.
sbin/md5: fix test case to not write outside temp directory
The bsd-p-test did create files 1.out .. 8.out in the test directory.
This has been fixed and the temporary output files are now written
to the temporary directory.
Rick Macklem [Fri, 11 Jun 2021 23:57:14 +0000 (16:57 -0700)]
krpc: Acquire ref count of CLIENT for backchannel use
Michael Dexter <editor@callfortesting.org> reported
a crash in FreeNAS, where the first argument to
clnt_bck_svccall() was no longer valid.
This argument is a pointer to the callback CLIENT
structure, which is free'd when the associated
NFSv4 ClientID is free'd.
This appears to have occurred because a callback
reply was still in the socket receive queue when
the CLIENT structure was free'd.
This patch acquires a reference count on the CLIENT
that is not CLNT_RELEASE()'d until the socket structure
is destroyed. This should guarantee that the CLIENT
structure is still valid when clnt_bck_svccall() is called.
It also adds a check for closed or closing to
clnt_bck_svccall() so that it will not process the callback
RPC reply message after the ClientID is free'd.
Rick Macklem [Wed, 9 Jun 2021 15:00:43 +0000 (08:00 -0700)]
nfscl: Add a "has acquired a delegation" flag for delegations
A problem was reported via email, where a large (130000+) accumulation
of NFSv4 opens on an NFSv4 mount caused significant lock contention
on the mutex used to protect the client mount's open/lock state.
Although the root cause for the accumulation of opens was not
resolved, it is obvious that the NFSv4 client is not designed to
handle 100000+ opens efficiently.
For a common case where delegations are not being issued by the
NFSv4 server, the code acquires the mutex lock for open/lock state,
finds the delegation list empty and just unlocks the mutex and returns.
This patch adds an NFS mount point flag that is set when a delegation
is issued for the mount. Then the patched code checks for this flag
before acquiring the open/lock mutex, avoiding the need to acquire
the lock for the case where delegations are not being issued by the
NFSv4 server.
This change appears to be performance neutral for a small number
of opens, but should reduce lock contention for a large number of opens
for the common case where server is not issuing delegations.
This commit should not affect the high level semantics of delegation
handling.
Rick Macklem [Mon, 7 Jun 2021 20:48:25 +0000 (13:48 -0700)]
nfscl: Fix generation of va_fsid for a tree of NFSv4 server file systems
Pre-r318997 the code looked like:
if (vp->v_mount->mnt_stat.f_fsid.val[0] != (uint32_t)np->n_vattr.na_filesid[0])
vap->va_fsid = (uint32_t)np->n_vattr.na_filesid[0];
Doing this assignment got lost by r318997 and, as such, NFSv4 mounts
of servers with trees of file systems on the server is broken, due to duplicate
fileno values for the same st_dev/va_fsid.
Although I could have re-introduced the assignment, since the value of
na_filesid[0] is not guaranteed to be unique across the server file systems,
I felt it was better to always do the hash for na_filesid[0,1].
Since dev_t (st_dev/va_fsid) is now 64bits, I switched to a 64bit hash.
There is a slight chance of a hash conflict where 2 different na_filesid
values map to same va_fsid, which will be documented in the BUGS
section of the man page for mount_nfs(8). Using a table to keep track
of mappings to catch conflicts would not easily scale to 10,000+ server file
systems and, when the conflict occurs, it only results in fts(3) reporting
a "directory cycle" under certain circumstances.
Rick Macklem [Sat, 5 Jun 2021 03:31:20 +0000 (20:31 -0700)]
nfsd: Fix NFSv4.1/4.2 Secinfo_no_name when security flavors empty
Commit 947bd2479ba9 added support for the Secinfo_no_name operation.
When a non-exported file system is being traversed, the list of
security flavors is empty. It turns out that the Linux client
mount attempt fails when the security flavors list in the
Secinfo_no_name reply is empty.
This patch modifies Secinfo/Secinfo_no_name so that it replies
with all four security flavors when the list is empty.
This fixes Linux NFSv4.1/4.2 mounts when the file system at
the NFSv4 root (as specified on a V4: exports(5) line) is
not exported.
Rick Macklem [Sat, 5 Jun 2021 23:53:07 +0000 (16:53 -0700)]
nfsd: Fix when NFSERR_WRONGSEC may be replied to NFSv4 clients
Commit d224f05fcfc1 pre-parsed the next operation number for
the put file handle operations. This patch uses this next
operation number, plus the type of the file handle being set by
the put file handle operation, to implement the rules in RFC5661
Sec. 2.6 with respect to replying NFSERR_WRONGSEC.
This patch also adds a check to see if NFSERR_WRONGSEC should be
replied when about to perform Lookup, Lookupp or Open with a file
name component, so that the NFSERR_WRONGSEC reply is done for
these operations, as required by RFC5661 Sec. 2.6.
This patch does not have any practical effect for the FreeBSD NFSv4
client and I believe that the same is true for the Linux client,
since NFSERR_WRONGSEC is considered a fatal error at this time.
Rick Macklem [Fri, 4 Jun 2021 03:48:26 +0000 (20:48 -0700)]
nfsd: Pre-parse the next NFSv4 operation number for put FH operations
RFC5661 Sec. 2.6 specifies when a NFSERR_WRONGSEC error reply can be done.
For the four operations PutFH, PutrootFH, PutpublicFH and RestoreFH,
NFSERR_WRONGSEC can or cannot be replied, depending upon what operation
follows one of these operations in the compound.
This patch modifies nfsrvd_compound() so that it parses the next operation
number before executing any of the above four operations, storing it in
"nextop".
A future commit will implement use of "nextop" to decide if NFSERR_WRONGSEC
can be replied for the above four operations.
This commit should not change the semantics of performing the compound RPC.
Rick Macklem [Wed, 2 Jun 2021 22:28:07 +0000 (15:28 -0700)]
nfsd: Fix the failure return for non-fh NFSv4 operations
Without this patch, nfsd_checkrootexp() returns failure
and then the NFSv4 operation would reply NFSERR_WRONGSEC.
RFC5661 Sec. 2.6 only allows a few NFSv4 operations, none
of which call nfsv4_checktootexp(), to return NFSERR_WRONGSEC.
This patch modifies nfsd_checkrootexp() to return the
error instead of a boolean and sets the returned error to an RPC
layer AUTH_ERR, as discussed on nfsv4@ietf.org.
The patch also fixes nfsd_errmap() so that the pseudo
error NFSERR_AUTHERR is handled correctly such that an RPC layer
AUTH_ERR is replied to the NFSv4 client.
The two new "enum auth_stat" values have not yet been assigned
by IANA, but are the expected next two values.
The effect on extant NFSv4 clients of this change appears
limited to reporting a different failure error when a
mount that does not use adequate security is attempted.
Rick Macklem [Tue, 1 Jun 2021 02:41:17 +0000 (19:41 -0700)]
nfsd: Delete extraneous NFSv4 root checks
There are several NFSv4.1/4.2 server operation functions which
have unneeded checks for the NFSv4 root being set up.
The checks are not needed because the operations always follow
a Sequence operation, which performs the check.
This patch deletes these checks, simplifying the code so
that a future patch that fixes the checks to conform with
RFC5661 Sec. 2.6 will be less extension.
Rick Macklem [Mon, 31 May 2021 00:52:43 +0000 (17:52 -0700)]
nfsd: Add support for the NFSv4.1/4.2 Secinfo_no_name operation
The Linux client is now attempting to use the Secinfo_no_name
operation for NFSv4.1/4.2 mounts. Although it does not seem to
mind the NFSERR_NOTSUPP reply, adding support for it seems
reasonable.
I also noticed that "savflag" needed to be 64bits in
nfsrvd_secinfo() since nd_flag in now 64bits, so I changed
the declaration of it there. I also added code to set "vp" NULL
after performing Secinfo/Secinfo_no_name, since these
operations consume the current FH, which is represented
by "vp" in nfsrvd_compound().
Fixing when the server replies NFSERR_WRONGSEC so that
it conforms to RFC5661 Sec. 2.6 still needs to be done
in a future commit.
Martin Matuska [Sat, 26 Jun 2021 11:37:50 +0000 (13:37 +0200)]
zfs: merge openzfs/zfs@aee26af27 (zfs-2.1-release) into stable/13
Notable upstream pull request merges:
#12172 Use wmsum for arc, abd, dbuf and zfetch statistics
#12227 Revert Consolidate arc_buf allocation checks
#12273 zfs_metaslab_mem_limit should be 25 instead of 75
#12266 Fix flag copying in resume case
#12276 Update cache file when setting compatibility property
Kristof Provost [Fri, 4 Jun 2021 08:52:07 +0000 (10:52 +0200)]
pf: store L4 headers in pf_pdesc
Rather than pointers to the headers store full copies. This brings us
slightly closer to what OpenBSD does, and also makes more sense than
storing pointers to stack variable copies of the headers.
Nathan Whitehorn [Fri, 28 May 2021 13:53:42 +0000 (09:53 -0400)]
Fix scripted installation from media without local distfiles.
The bsdinstall script target did not have the infrastructure to fetch
distfiles from a remote server the way the interactive installer does
on e.g. bootonly media. Solve this by factoring out the parts of the
installer that deal with fetching missing distributions into a new
install stage called 'fetchmissingdists', which is called by both the
interactive and scripted installer frontends.
In the course of these changes, cleaned up a few other issues with
the fetching of missing distribution files and added a warning if
fetching the MANIFEST file, which is used to verify the integrity of
the distribution files. We should at some point add cryptographic
signatures to MANIFEST so that it can be fetched safely if not present
on the install media (which it is for bootonly media).
Dimitry Andric [Fri, 25 Jun 2021 18:42:38 +0000 (20:42 +0200)]
Work around bogus old gcc "initializer element is not constant" error
After df3b437c1e073eb83e9a93af1c417f3ee8d0de3b, older gcc's such as
4.2.1 (still used on earlier branches for e.g. mips and powerpc) and
6.3.0 (still used for some cross-builds) started throwing bogus errors
like:
In file included from /workspace/src/lib/msun/src/s_llround.c:11:0:
/workspace/src/lib/msun/src/s_lround.c:54:31: error: initializer element is not constant
static const type dtype_min = type_min - 0.5;
^~~~~~~~
/workspace/src/lib/msun/src/s_lround.c:55:31: error: initializer element is not constant
static const type dtype_max = type_max + 0.5;
^~~~~~~~
Since 'type_min' and 'type_max' are constants declared just above these
lines this error is nonsensical, but older gcc's are not smart enough.
Work around the error by reusing the (type)DTYPE_MIN and (type)DTYPE_MAX
macros, so I can MFC this right away, unbreaking a few stable builds.
Dimitry Andric [Tue, 22 Jun 2021 16:38:27 +0000 (18:38 +0200)]
Fix failures in libm's lround_test after clang 12 import
It turned out that the (type)DTYPE_MAX conversions at the top of
s_lround.c are now emitted as cvtsi2sd instructions, at least on SSE
capable CPUs. This caused the FE_INEXACT flag to always be set, at least
for the double and float variants. Under clang 11, the whole INRANGE()
comparisons were still optimized away, but this has "improved" in clang
12, due to stricter adherence to the -ffp-exception-behavior=maytrap
compiler flag.
To avoid run-time integer to float conversions, use static constants
instead, so they are computed at compile time, and the INRANGE()
statements are optimized away again, if applicable.
While here, use an integer instead of a floating type to store the test
results in lround_test.c, as this is more appropriate, and we can also
drop the volatile hack.
Dimitry Andric [Tue, 22 Jun 2021 20:26:13 +0000 (22:26 +0200)]
Disable llvm generating 128-bit multiply libcalls on 32-bit ARM
Merge commit 789708617d20 from llvm git (Koutheir Attouchi):
Do not generate calls to the 128-bit function __multi3() on 32-bit ARM
Re-applying this patch after bots failures. Should be fine now.
The function __multi3() is undefined on 32-bit ARM, so a call to it should
never be emitted. Instead, plain instructions need to be generated to
perform 128-bit multiplications.
Mitchell Horne [Mon, 1 Mar 2021 15:07:54 +0000 (11:07 -0400)]
Consolidate machine/endian.h definitions
This change serves two purposes.
First, we take advantage of the compiler provided endian definitions to
eliminate some long-standing duplication between the different versions
of this header. __BYTE_ORDER__ has been defined since GCC 4.6, so there
is no need to rely on platform defaults or e.g. __MIPSEB__ to determine
endianness. A new common sub-header is added, but there should be no
changes to the visibility of these definitions.
Second, this eliminates the hand-rolled __bswapNN() routines, again in
favor of the compiler builtins. This was done already for x86 in e6ff6154d203. The benefit here is that we no longer have to maintain our
own implementations on each arch, and can instead rely on the compiler
to emit appropriate instructions or libcalls, as available. This should
result in equivalent or better code generation. Notably 32-bit arm will
start using the `rev` instruction for these routines, which is available
on armv6+.
Brian Behlendorf [Thu, 24 Jun 2021 21:30:02 +0000 (14:30 -0700)]
Update cache file when setting compatibility property
Unlike most other properties the 'compatibility' property is stored
in the pool config object and not the DMU_OT_POOL_PROPS object.
This had the advantage that the compatibility information is available
without needing to fully import the pool (it can be read with zdb).
However, this means we need to make sure to update both the copy of
the config in the MOS and the cache file. This wasn't being done.
This commit adds a call to spa_async_request() to ensure the copy of
the config in the cache file gets updated as well as the one stored
in the pool. This same change is made for the 'comment' property
which suffers from the same inconsistency.
Reviewed-by: Sean Eric Fagan <sef@ixsystems.com> Reviewed-by: Colm Buckley <colm@tuatha.org> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #12261
Closes #12276
Paul Dagnelie [Thu, 24 Jun 2021 19:42:01 +0000 (12:42 -0700)]
Fix flag copying in resume case
A couple flags weren't being copied in the case where we're doing size
estimation on a resume.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Matthew Ahrens <mahrens@delphix.com> Signed-off-by: Paul Dagnelie <pcd@delphix.com> Closes: #12266
jumbi77 [Thu, 24 Jun 2021 17:02:54 +0000 (19:02 +0200)]
zfs_metaslab_mem_limit should be 25 instead of 75
According to current zfs man page zfs_metaslab_mem_limit should be
25 instead of 75.
Reviewed-by: Matthew Ahrens <mahrens@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Paul Dagnelie <pcd@delphix.com> Reviewed-by: Mark Maybee <mark.maybee@delphix.com> Signed-off-by: jumbi77@users.noreply.github.com
Closes #12273
Attila Fülöp [Wed, 23 Jun 2021 23:57:06 +0000 (01:57 +0200)]
gcc 11 cleanup
Compiling with gcc 11.1.0 produces three new warnings.
Change the code slightly to avoid them.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Matthew Ahrens <mahrens@delphix.com> Signed-off-by: Attila Fülöp <attila@fueloep.org>
Closes #12130
Closes #12188
Closes #12237
Brian Behlendorf [Wed, 23 Jun 2021 22:53:13 +0000 (15:53 -0700)]
ZTS: Add known exceptions
The receive-o-x_props_override test case reliably fails on the
FreeBSD main builders (but not on Linux), until the root cause is
understood add this test to the FreeBSD exception list.
On Linux the alloc_class_012_pos test case may occasionally fail.
This is a known false positive which has also been added to the
Linux exception list until the test can be made entirely reliable.
Reviewed-by: George Melikov <mail@gmelikov.ru> Reviewed-by: John Kennedy <john.kennedy@delphix.com> Reviewed-by: Ryan Moeller <ryan@iXsystems.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #12272
Rich Ercolani [Wed, 23 Jun 2021 04:53:45 +0000 (00:53 -0400)]
Annotated dprintf as printf-like
ZFS loves using %llu for uint64_t, but that requires a cast to not
be noisy - which is even done in many, though not all, places.
Also a couple places used %u for uint64_t, which were promoted
to %llu.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Rich Ercolani <rincebrain@gmail.com>
Closes #12233
Per the discussion in #11531, the reverted commit---which intended only
to be a cleanup commit---introduced a subtle, unintended change in
behavior.
Care was taken to partially revert and then reapply 10b3c7f5e4
which would otherwise have caused a conflict. These changes were
squashed in to this commit.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Suggested-by: @chrisrd Suggested-by: robn@despairlabs.com Signed-off-by: Antonio Russo <aerusso@aerusso.net>
Closes #11531
Closes #12227
Alexander Motin [Thu, 17 Jun 2021 00:19:34 +0000 (20:19 -0400)]
Use wmsum for arc, abd, dbuf and zfetch statistics. (#12172)
wmsum was designed exactly for cases like these with many updates
and rare reads. It allows to completely avoid atomic operations on
congested global variables.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Mark Maybee <mark.maybee@delphix.com> Signed-off-by: Alexander Motin <mav@FreeBSD.org> Sponsored-By: iXsystems, Inc.
Closes #12172
This replaces the generic libspl atomic.c atomics implementation
with one based on builtin gcc atomics. This functionality was added
as an experimental feature in gcc 4.4. Today even CentOS 7 ships
with gcc 4.8 as the default compiler we can make this the default.
Furthermore, the builtin atomics are as good or better than our
hand-rolled implementation so it's reasonable to drop that custom code.
Mark Johnston [Tue, 8 Jun 2021 13:40:30 +0000 (09:40 -0400)]
hyperv: Fix vmbus after the i386 4/4 split
The vmbus ISR needs to live in a trampoline. Dynamically allocating a
trampoline at driver initialization time poses some difficulties due to
the fact that the KENTER macro assumes that the offset relative to
tramp_idleptd is fixed at static link time. Another problem is that
native_lapic_ipi_alloc() uses setidt(), which assumes a fixed trampoline
offset.
Rather than fight this, move the Hyper-V ISR to i386/exception.s. Add a
new HYPERV kernel option to make this optional, and configure it by
default on i386. This is sufficient to make use of vmbus(4) after the
4/4 split. Note that vmbus cannot be loaded dynamically and both the
HYPERV option and device must be configured together. I think this is
not too onerous a requirement, since vmbus(4) was previously
non-functional.
Reported by: Harry Schmalzbauer <freebsd@omnilan.de>
Tested by: Harry Schmalzbauer <freebsd@omnilan.de>
Reviewed by: whu, kib
Sponsored by: The FreeBSD Foundation
Dimitry Andric [Mon, 14 Jun 2021 19:17:05 +0000 (21:17 +0200)]
Export various 128 bit long double functions from libgcc_s.so.1
These were already compiled for some time on aarch64 and riscv, by
including lib/libcompiler_rt/Makefile.inc, but never exported in the
shared library. Since gcc exports these under version GCC_4.6.0, we do
the same.
This review should replace D11482 for now. For e.g. amd64 more work is
still to be done, as compiler-rt does not seem to support 128 bit long
double math for that architecture.
Reviewed by: cem
Differential Revision: https://reviews.freebsd.org/D28690
Mark Johnston [Mon, 14 Jun 2021 21:32:27 +0000 (17:32 -0400)]
Consistently use the SOLISTENING() macro
Some code was using it already, but in many places we were testing
SO_ACCEPTCONN directly. As a small step towards fixing some bugs
involving synchronization with listen(2), make the kernel consistently
use SOLISTENING(). No functional change intended.
Mark Johnston [Mon, 14 Jun 2021 21:32:18 +0000 (17:32 -0400)]
amd64: Fix propagation of LDT updates
When a process has used sysarch(2) to specify descriptors for its
private LDT, upon rfork(RFMEM) descriptors are copied into the new child
process. Any updates to the descriptors are thus reflected to all other
processes sharing the vmspace. However, this is incorrect in the rather
obscure case where the child process was created before the LDT was
modified. Fix this by only modifying other processes which already
share the LDT.
Reported by: syzkaller
Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Marko Zec [Thu, 17 Jun 2021 06:49:09 +0000 (08:49 +0200)]
tests: Revise FIB lookups per second benchmarking routines
Fix a bug in the LPM SEQ benchmark (missing break inside a switch block)
by restructuring the test loop, while introducing additional two
synthetic test options:
ANN: scan only the address space announced in current RIB
REP: repeat lookups over several keys in a sliding window scheme
The total of eight combinations of test options are now available
through dedicated sysctl hooks.
Differential Revision: <https://reviews.freebsd.org/D30311>
Reviewed by: melifaro
MFC after: 3 days
Marko Zec [Wed, 5 May 2021 10:28:17 +0000 (12:28 +0200)]
Revise FIB lookups per second benchmarking routines.
Add a LPS benchmark variant which introduces artificial dependencies
between successive lookups. While here, instead of writing the results
from the lookups to a huge array, add them to an accumulator, in a more
lightweight attempt at preventing the CPU's OOO machinery from
discarding the lookup results if they would be completely unused.
net.route.test.run_lps_rnd measures LPS throughput with independent
uniformly random keys
net.route.test.run_lps_seq measures LPS throughput with uniformly
random keys with artificial interdependencies
Reviewed by: melifaro
MFC after: 7 days
Differential Revision: https://reviews.freebsd.org/D30096
Mark Johnston [Wed, 16 Jun 2021 13:46:56 +0000 (09:46 -0400)]
ipfw: Update the pfil mbuf pointer in ipfw_check_frame()
ipfw_chk() might call m_pullup() and thus can change the mbuf chain
head. In this case, the new chain head has to be returned to the pfil
hook caller, otherwise the pfil hook caller is left with a dangling
pointer.
Note that this affects only the link-layer hooks installed when the
net.link.ether.ipfw sysctl is set to 1.
PR: 256439, 254015, 255069, 255104
Fixes: f355cb3e6
Reviewed by: ae
Sponsored by: The FreeBSD Foundation
George Amanakis [Thu, 17 Jun 2021 00:17:42 +0000 (03:17 +0300)]
Avoid deadlock when removing L2ARC devices under I/O
In case we have I/O and try to remove an L2ARC device a deadlock might
occur. arc_read()->zio_read()->zfs_blkptr_verify() waits for SCL_VDEV
to be dropped while holding the hash_lock. However, spa_l2cache_load()
holds SCL_ALL and waits for the hash_lock in l2arc_evict().
Fix this by moving zfs_blkptr_verify() to the top top arc_read() before
the hash_lock is taken. Verify the block pointer and return a checksum
error if damaged rather than halting the system, by using
BLK_VERIFY_LOG instead of BLK_VERIFY_HALT.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Mark Maybee <mark.maybee@delphix.com> Signed-off-by: George Amanakis <gamanakis@gmail.com>
Closes #12054
Alan Somers [Thu, 22 Apr 2021 21:09:03 +0000 (15:09 -0600)]
gmultipath: make physpath distinct from the underlying providers'
zfsd uses a device's physical path attribute to automatically replace a
missing ZFS disk when a blank disk is inserted into the same physical
slot. Currently gmultipath passes through its underlying providers'
physical path attribute. That may cause zfsd to replace a missing
gmultipath provider with a newly arrived, single-path disk. That would
be bad.
This commit fixes that problem by simply appending "/mp" to the
underlying providers' physical path, in a manner similar to what geli
already does.
Kristof Provost [Fri, 21 May 2021 12:26:49 +0000 (14:26 +0200)]
dummynet: Fix schedlist and aqmlist locking
These are global (i.e. shared across vnets) structures, so we need
global lock to protect them. However, we look up entries in these lists
(find_aqm_type(), find_sched_type()) and return them. We must ensure
that the returned structures cannot go away while we are using them.
Resolve this by using NET_EPOCH(). The structures can be safely accessed
under it, and we postpone their cleanup until we're sure they're no
longer used.
Marko Zec [Wed, 5 May 2021 11:45:52 +0000 (13:45 +0200)]
Introduce DXR as an IPv4 longest prefix matching / FIB module
DXR maintains compressed lookup structures with a trivial search
procedure. A two-stage trie is indexed by the more significant bits of
the search key (IPv4 address), while the remaining bits are used for
finding the next hop in a sorted array. The tradeoff between memory
footprint and search speed depends on the split between the trie and
the remaining binary search. The default of 20 bits of the key being
used for trie indexing yields good performance (see below) with
footprints of around 2.5 Bytes per prefix with current BGP snapshots.
Rebuilding lookup structures takes some time, which is compensated for by
batching several RIB change requests into a single FIB update, i.e. FIB
synchronization with the RIB may be delayed for a fraction of a second.
RIB to FIB synchronization, next-hop table housekeeping, and lockless
lookup capability is provided by the FIB_ALGO infrastructure.
DXR works well on modern CPUs with several MBytes of caches, especially
in VMs, where is outperforms other currently available IPv4 FIB
algorithms by a large margin.
Test functionality of ng_vlan_rotate(4):
- Rotate 1 to 9 stagged vlans in any possible direction and length
- Rotate random combinations of ethertypes (8100, 88a8, 9100)
- Automatic reverse rotating for backward data flow
- Test too many and too few vlans
Test functionality of ng_hub(4):
- replicting traffic to anything but the sending hook
- persistence
- an unrestricted loop
- implementation limits with many hooks.
Turns out $ZPOOL_IMPORT_OPTS expands in a shell-like fashion,
yielding 'import' '-aN' '-o' 'cachefile=none' for an unset variable,
and 'import' '-aN' '-o' 'cachefile=none' 'word1' 'word2' for a
white-spaced one, but ${ZPOOL_IMPORT_OPTS} expands like "${Z_I_O}"
would in a shell, yielding 'import' '-aN' '-o' 'cachefile=none' ''
(empty) and 'import' '-aN' '-o' 'cachefile=none' 'word1 word2' (spaced)
Matthew Ahrens [Sun, 13 Jun 2021 17:48:53 +0000 (10:48 -0700)]
vdev_draid_min_asize() ignores reserved space
vdev_draid_min_asize() returns the minimum size of a child vdev. This
is used when determining if a disk is big enough to replace a child.
It's also used by zdb to determine how big of a child to make to test
replacement.
vdev_draid_min_asize() says that the child’s asize has to be at least
1/Nth of the entire draid’s asize, which is the same logic as raidz.
However, this contradicts the code in vdev_draid_open(), which
calculates the draid’s asize based on a reduced child size:
An additional 32MB of scratch space is reserved at the end of each
child for use by the dRAID expansion feature
So the problem is that you can replace a draid disk with one that’s
vdev_draid_min_asize(), but it actually needs to be larger to accommodate
the additional 32MB. The replacement is allowed and everything works at
first (since the reserved space is at the end, and we don’t try to use
it yet), but when you try to close and reopen the pool,
vdev_draid_open() calculates a smaller asize for the draid, because of
the smaller leaf, which is not allowed.
I think the confusion is that vdev_draid_min_asize() is correctly
returning the amount of required *allocatable* space in a leaf, but the
actual *size* of the leaf needs to be at least 32MB more than that.
ztest_vdev_attach_detach() assumes that it can attach that size of
device, and it actually can (the kernel/libzpool accepts it), but it
then later causes zdb to not be able to open the pool.
This commit changes vdev_draid_min_asize() to return the required size
of the leaf, not the size that draid will make available to the metaslab
allocator.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Mark Maybee <mark.maybee@delphix.com> Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #11459
Closes #12221
Paul Zuchowski [Sat, 12 Jun 2021 00:00:33 +0000 (20:00 -0400)]
Do not hash unlinked inodes
In zfs_znode_alloc we always hash inodes. If the
znode is unlinked, we do not need to hash it. This
fixes the problem where zfs_suspend_fs is doing zrele
(iput) in an async fashion, and zfs_resume_fs unlinked
drain processing will try to hash an inode that could
still be hashed, resulting in a panic.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Alan Somers <asomers@gmail.com> Signed-off-by: Paul Zuchowski <pzuchowski@datto.com>
Closes #9741
Closes #11223
Closes #11648
Closes #12210
Brian Behlendorf [Fri, 11 Jun 2021 15:21:36 +0000 (08:21 -0700)]
ZTS: Add zfs_clone_livelist_dedup.ksh to Makefile.am
Commit 86b5f4c12 added a new zfs_clone_livelist_dedup.ksh test case
but didn't include it in the Makefile.am. This results in the test
not being included in the dist tarball so it's never run by the CI.
Reviewed-by: John Kennedy <john.kennedy@delphix.com> Reviewed-by: Serapheim Dimitropoulos <serapheim@delphix.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes: #12224
Pedro F. Giffuni [Mon, 31 May 2021 01:48:38 +0000 (20:48 -0500)]
fread: improve performance for unbuffered reads
We can use the buffer passed to fread(3) directly in the FILE *.
The buffer needs to be reset before each call to __srefill().
This preserves the expected behavior in all cases.
The change was found originally in OpenBSD and later adopted by NetBSD.
Casper services expect that the first 3 descriptors (stdin/stdout/stderr)
will point to /dev/null. Which Casper will ensure later. The Casper
services are forked from the original process. If the initial process
closes one of those descriptors, Casper may reuse one of them for it on
purpose. If this is the case, then renumarate the descriptors used by
Casper to higher numbers. This is done already after the fork, so it
doesn't break the parent process.
Test functionality of ng_bridge(4):
- replicating traffic to anything but the sending hook
- persistence
- detect loops
- unicast to only one link of many
- stretch to implementation limits on broadcast
tests/netgraph: Inital framework for testing libnetgraph
Provide a framework of functions to test various netgraph modules.
Tests contain:
- creating, renaming, and destroying nodes
- connecting and removing hooks
- sending and receiving data
- sending ASCII messages and receiving binary responses
- errors can be passed for indiviual inspection or fail the test
Also contains some fixups:
- indent all files correctly
- finish factoring out
- remove debugging code
- check for renaming issues reported in PR241954
Randall Stewart [Fri, 11 Jun 2021 15:38:08 +0000 (11:38 -0400)]
tcp: Missing mfree in rack and bbr
Recently (Nov) we added logic that protects against a peer negotiating a timestamp, and
then not including a timestamp. This involved in the input path doing a goto done_with_input
label. Now I suspect the code was cribbed from one in Rack that has to do with the SYN.
This had a bug, i.e. it should have a m_freem(m) before going to the label (bbr had this
missing m_freem() but rack did not). This then caused the missing m_freem to show
up in both BBR and Rack. Also looking at the code referencing m->m_pkthdr.lro_nsegs
later (after processing) is not a good idea, even though its only for logging. Best to
copy that off before any frees can take place.
Randall Stewart [Thu, 10 Jun 2021 12:33:57 +0000 (08:33 -0400)]
tcp: Mbuf leak while holding a socket buffer lock.
When running at NF the current Rack and BBR changes with the recent
commits from Richard that cause the socket buffer lock to be held over
the ip_output() call and then finally culminating in a call to tcp_handle_wakeup()
we get a lot of leaked mbufs. I don't think that this leak is actually caused
by holding the lock or what Richard has done, but is exposing some other
bug that has probably been lying dormant for a long time. I will continue to
look (using his changes) at what is going on to try to root cause out the issue.
In the meantime I can't leave the leaks out for everyone else. So this commit
will revert all of Richards changes and move both Rack and BBR back to just
doing the old sorwakeup_locked() calls after messing with the so_rcv buffer.
We may want to look at adding back in Richards changes after I have pinpointed
the root cause of the mbuf leak and fixed it.
Randall Stewart [Wed, 9 Jun 2021 17:58:54 +0000 (13:58 -0400)]
tcp: LRO timestamps have lost their previous precision
Recently we had a rewrite to tcp_lro.c that was tested but one subtle change
was the move to a less precise timestamp. This causes all kinds of chaos
in tcp's that do pacing and needs to be fixed to use the more precise
time that was there before.
projects / FreeBSD / FreeBSD.git / log