Alexander Motin [Wed, 4 Mar 2020 04:44:08 +0000 (04:44 +0000)]
MFC r358357: MFZoL: Relax restriction on zfs_ioc_next_obj() iteration
Per the documentation for dnode_next_offset in dnode.c, the "txg"
parameter specifies a lower bound on which transaction the dnode can
be found in. We are interested in all dnodes that are removed between
the first and last transaction in the snapshot. It doesn't need to be
created in that snapshot to correspond to a removed file.
In fact, the behavior of zfs diff in the test case exactly matches
this: the transaction that created the data that was deleted in snapshot
"2" was produced before, in snapshot "1", definitely predating the first
transaction in snapshot "2".
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tim Chase <Tim Chase <tim@onlight.com>
Closes #2081
zfsonlinux/zfs@7290cd3c4ed19fb3f75b8133db2e36afcdd24beb
Alexander Motin [Wed, 4 Mar 2020 04:42:56 +0000 (04:42 +0000)]
MFC r358342: MFZoL: Fix resilver writes in vdev_indirect_io_start
This patch addresses an issue found in ztest where resilver
write zios that were passed to an indirect vdev would end up
being handled as though they were resilver read zios. This
caused issues where the zio->io_abd would be both read to
and written from at the same time, causing asserts to fail.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Matt Ahrens <matt@delphix.com> Reviewed-by: Serapheim Dimitropoulos <serapheim@delphix.com> Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8193
zfsonlinux/zfs@5aa95ba0d3502779695341b5f55fa5ba1d3330ff
Alexander Motin [Wed, 4 Mar 2020 04:41:49 +0000 (04:41 +0000)]
MFC r358339: MFZoL: Fix issue with scanning dedup blocks as scan ends
This patch fixes an issue discovered by ztest where
dsl_scan_ddt_entry() could add I/Os to the dsl scan queues
between when the scan had finished all required work and
when the scan was marked as complete. This caused the scan
to spin indefinitely without ending.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Serapheim Dimitropoulos <serapheim.dimitro@delphix.com> Reviewed-by: Matthew Ahrens <mahrens@delphix.com> Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8010
zfsonlinux/zfs@5e0bd0ae056e26de36dee3c199c6fcff8f14ee15
Alexander Motin [Wed, 4 Mar 2020 04:40:35 +0000 (04:40 +0000)]
MFC r358337: MFZoL: Fix 2 small bugs with cached dsl_scan_phys_t
This patch corrects 2 small bugs where scn->scn_phys_cached was
not properly updated to match the primary copy when it needed to
be. The first resulted in the pause state not being properly
updated and the second resulted in the cached version being
completely zeroed even if the primary was not.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Serapheim Dimitropoulos <serapheim.dimitro@delphix.com> Reviewed-by: Matthew Ahrens <mahrens@delphix.com> Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #8010
zfsonlinux/zfs@8cb119e3dc0ac6c90b1517fbadc021b7e9741fc6
Alexander Motin [Wed, 4 Mar 2020 04:36:50 +0000 (04:36 +0000)]
MFC r358336: MFZoL: Fix txg_sync_thread hang in scan_exec_io()
When scn->scn_maxinflight_bytes has not been initialized it's
possible to hang on the condition variable in scan_exec_io().
This issue was uncovered by ztest and is only possible when
deduplication is enabled through the following call path.
Resolve the issue by always initializing scn_maxinflight_bytes
to a reasonable minimum value. This value will be recalculated
in dsl_scan_sync() to pick up changes to zfs_scan_vdev_limit
and the addition/removal of vdevs.
Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed by: George Melikov <mail@gmelikov.ru> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #7098
zfsonlinux/zfs@f90a30ad1b32a971f62a540f8944e42f99b254ce
Dimitry Andric [Tue, 3 Mar 2020 18:01:03 +0000 (18:01 +0000)]
MFC r358407:
Merge r358406 from the clang1000-import branch:
Fix the following -Werror warning from clang 10.0.0:
sys/arm/arm/identcpu-v6.c:227:5: error: misleading indentation; statement is not part of the previous 'if' [-Werror,-Wmisleading-indentation]
if (val & CPUV7_CT_CTYPE_RA)
^
sys/arm/arm/identcpu-v6.c:225:4: note: previous statement is here
if (val & CPUV7_CT_CTYPE_WB)
^
This was due to an accidentally inserted tab before the if statement.
Bjoern A. Zeeb [Mon, 2 Mar 2020 22:54:32 +0000 (22:54 +0000)]
MFC r358297: Fix IPv6 checksums when exthdrs are present.
In two places in ip6_output we are doing (delayed) checksum calculations.
The initial logic came from SCTP in r205075,205104 and later I copied
and adjusted it for the TCP|UDP case in r235958.
The problem was that the original SCTP offsets were already wrong for any
case with extension headers present given IPv6 extension headers are not
part of the pseudo checksum calculations.
The later changes do not help in case there is checksum offloading as for
extension headers (incl. fragments) we do currrently never offload as we
have no infrastructure to know whether the NIC can handle these cases.
Correct the offsets for delayed checksum calculations and properly handle
mbuf flags. In addition harmonize the almost identical duplicate code.
While here eliminate the now unneeded variable hlen and add an always
missing mtod() call in the 1-b and 3 cases after the introduction of
the mb_unmapped_to_ext() calls. [Keep code in sync with head]
Reported by: Francis Dupont (fdupont isc.org)
PR: 243675
Bjoern A. Zeeb [Mon, 2 Mar 2020 13:23:28 +0000 (13:23 +0000)]
MFC r358427: initialize oifp to avoid bogus results/panics in edge cases
In certain cases (probably not during normal operation but observed in
the lab during development) ip6_ouput() could return without error
and ifpp (&oifp) not updated.
Given oifp was never initialized we would take the later branch
as oifp was not NULL, and when calling icmp6_ifstat_inc() we would
panic dereferencing a garbage pointer.
For code stability initialize oifp to NULL before first use to always
have a deterministic value and not rely on a called function to behave
and always and for ever do the work for us as we hope for.
Move IPv6 source address checks from after extension header heandling
to the top of the function. If we do not pass these checks there is
no reason to do a lot of work upfront.
Fold extension header preparations and length calculations together into
a single branch and macro rather than doing them sequentially.
Likewise move extension header concatination into a single branch block
only doing it if we recorded any extension header length length.
MFC r358219 and r358270:
Make sure the ACPI lid state is updated during boot and after resume.
While at it update the sysctl(9) description for the lid state.
Always check return value from acpi_GetInteger().
If a failure happens reading the lid state, assume the lid is opened.
Dimitry Andric [Mon, 2 Mar 2020 07:11:37 +0000 (07:11 +0000)]
MFC r358132:
Take LINKER_FREEBSD_VERSION from numerical field after dash
Summary:
With COMPILER_FREEBSD_VERSION, we use a numeric value that we bump each
time we make a change that requires re-bootstrapping, but with the
linker variant, we instead take the entire part after "FreeBSD", as in
this example version output:
We should only look at the numerical field we append after a dash
instead. This review attempts to make it so.
The only thing I am not happy about is the post-processing of awk output
in Makefile.inc1. I notice that our awk does not have gensub(), so it
can't substitute a numbered sub-regex with \1, \2, etc. Suggestions
welcome. :)
MFC r358099:
Add some HISTORY sections to manpages
environ(7) was in AT&T Version 7
ac(8): Add a HISTORY section
sa(8): Add a HISTORY section
sqrt(3): Add the actual sqrt function to the HISTORY section
MFC r358310:
Use hid_get_data_unsigned() instead of hid_get_data() when reading the
key-codes from the USB keyboard. Negative key-codes are currently skipped.
While at it use the bit size value provided by the HID location structure
instead of assuming a value of 8.
Kyle Evans [Fri, 28 Feb 2020 18:34:17 +0000 (18:34 +0000)]
MFC r358091: certctl(8): switch to install(1) to fix DESTDIR support
"Oops" - ln(1) is fine and dandy, but when you're using DESTDIR...it's not-
the path will almost certainly be invalid once the root you've just
installed to is relocated, perhaps to /.
Switch to install(1) using `-l rs` to calculate the relative symlink between
the two, which should work just fine in all cases.
bhyve: enable virtio-net mergeable rx buffers for tap(4)
This patch adds a new netbe_peek_recvlen() function to the net
backend API. The new function allows the virtio-net receive code
to know in advance how many virtio descriptors chains will be
needed to receive the next packet. As a result, the implementation
of the virtio-net mergeable rx buffers feature becomes efficient,
so that we can enable it also with the tap(4) backend. For the
tap(4) backend, a bounce buffer is introduced to implement the
peeck_recvlen() callback, which implies an additional packet copy
on the receive datapath. In the future, it should be possible to
remove the bounce buffer (and so the additional copy), by
obtaining the length of the next packet from kevent data.
Andriy Gapon [Thu, 27 Feb 2020 15:08:43 +0000 (15:08 +0000)]
MFC r357042: vmxnet3: add support for RSS kernel option
We observe at least one problem: if a UDP socket is connect(2)-ed, then a
received packet that matches the connection cannot be matched to the
corresponding PCB because of an incorrect flow ID. That was oberved for DNS
requests from the libc resolver. We got this problem because FreeBSD
r343291 enabled code that can set rsstype of received packets to values
other than M_HASHTYPE_OPAQUE_HASH. Earlier that code was under 'ifdef
notyet'.
The essence of this change is to use the system-wide RSS key instead of
some historic hardcoded key when the software RSS is enabled and it is
configured to use Toeplitz algorithm (the default).
In all other cases, the driver reports the opaque hash type for received
packets while still using Toeplitz algorithm with the internal key.
Andriy Gapon [Thu, 27 Feb 2020 14:52:55 +0000 (14:52 +0000)]
MFC r357040: virtio_scsi: use max target ID plus one as the initiator ID
This bus does not really have a concept of the initiator ID, so use
a guaranteed dummy one that won't conflict with any real target.
This change fixes a problem with virtio_scsi on GCE where disks get
sequential target IDs starting from one. If there are seven or more
disks, then a disk with the target ID of seven would not be discovered
by FreeBSD as that ID was reserved as the initiator ID -- see
scsi_scan_bus().
Andriy Gapon [Thu, 27 Feb 2020 14:50:02 +0000 (14:50 +0000)]
MFC r356730: storvsc: properly set residual data length on errors
This change is based on Linux commit 40630f462824ee. csio.resid should
account for transfer_len only for success and SRB_STATUS_DATA_OVERRUN
condition.
I am not sure how exactly this change works, but I have a report from a
user that they see lots of checksum errors when running a pool scrub
concurrently with iozone -l 1 -s 100G. After applying this patch the
problem cannot be reproduced.
Andriy Gapon [Thu, 27 Feb 2020 14:48:12 +0000 (14:48 +0000)]
MFC r355736: zfs boot: fix a crash in a rarely taken path in fzap_lookup
Instead of passing NULL to fzap_name_equal and crashing, just return
ENOENT. This happened when higher bits of a hash of the searched key
(its hash prefix) matched a hash prefix of some key in the ZAP, but the
full hash value of the searched key did not match any key in the ZAP.
I observerved this problem when loader tried to look up
"features_for_read" in a particular old pool that predates pool
features.
Cy Schubert [Tue, 25 Feb 2020 21:02:52 +0000 (21:02 +0000)]
MFC r358064:
As with ipf(8), give ippool(8) the ability to load IP pools from multiple
files. This allows for loading, during the same invocation of ippool, of
multiple sources of input using multiple tools to concurrently maintain the
files such as fail2ban, macro preprocessors, and manually.
Alexander Motin [Tue, 25 Feb 2020 15:52:35 +0000 (15:52 +0000)]
MFC r349381: Avoid extra taskq_dispatch() calls by DMU.
DMU sync code calls taskq_dispatch() for each sublist of os_dirty_dnodes
and os_synced_dnodes. Since the number of sublists by default is equal
to number of CPUs, it will dispatch equal, potentially large, number of
tasks, waking up many CPUs to handle them, even if only one or few of
sublists actually have any work to do.
This change adds check for empty sublists to avoid this.
MFC r358013:
Fix kernel panic while trying to read multicast stream.
When VIMAGE is enabled make sure the "m_pkthdr.rcvif" pointer is set
for all mbufs being input by the IGMP/MLD6 code. Else there will be a
NULL-pointer dereference in the netisr code when trying to set the
VNET based on the incoming mbuf. Add an assert to catch this when
queueing mbufs on a netisr to make debugging of similar cases easier.
Found by: Vladislav V. Prodan
PR: 244002
Reviewed by: bz@
Sponsored by: Mellanox Technologies
MFC r357799:
Make sure the so-called end of receive interrupts don't starve in iflib.
When the receive ring cannot be filled with mbufs, due to lack of memory,
no more interrupts may be generated to fill the receive ring later on.
Make sure to have a watchdog, to try refilling the receive ring from time
to time, hopefully when more mbufs are available.
Kyle Evans [Sat, 22 Feb 2020 21:44:00 +0000 (21:44 +0000)]
MFC r358109: kdump: decode SHM_ANON as first arg to legacy shm_open(2)
The first argument to shm_open(2) as well as shm_open2(2) may be a path or
SHM_ANON. Decode SHM_ANON, at least- paths will show up as namei results in
kdump output, which may be sufficient; in those cases, we'll have printed an
address.
Dimitry Andric [Fri, 21 Feb 2020 22:11:14 +0000 (22:11 +0000)]
MFC r358074:
Fix the following -Werror warning from clang 10.0.0:
sys/arm/allwinner/clkng/aw_clk_mipi.c:144:6: error: misleading indentation; statement is not part of the previous 'if' [-Werror,-Wmisleading-indentation]
m++;
^
sys/arm/allwinner/clkng/aw_clk_mipi.c:142:5: note: previous statement is here
if (best == *fout)
^
Move the increment operations into the for loop headers instead.
Dimitry Andric [Fri, 21 Feb 2020 21:57:24 +0000 (21:57 +0000)]
MFC r358046:
Merge r358042 from the clang1000-import branch:
Add casts and L suffixes to libc quad support, to work around various
-Werror warnings from clang 10.0.0, such as:
lib/libc/quad/fixdfdi.c:57:12: error: implicit conversion from 'long long' to 'double' changes value from 9223372036854775807 to 9223372036854775808 [-Werror,-Wimplicit-int-float-conversion]
if (x >= QUAD_MAX)
~~ ^~~~~~~~
/usr/obj/usr/src/powerpc.powerpc/tmp/usr/include/sys/limits.h:89:19: note: expanded from macro 'QUAD_MAX'
#define QUAD_MAX (__QUAD_MAX) /* max value for a quad_t */
^~~~~~~~~~
/usr/obj/usr/src/powerpc.powerpc/tmp/usr/include/machine/_limits.h:91:20: note: expanded from macro '__QUAD_MAX'
#define __QUAD_MAX __LLONG_MAX /* max value for a quad_t */
^~~~~~~~~~~
/usr/obj/usr/src/powerpc.powerpc/tmp/usr/include/machine/_limits.h:75:21: note: expanded from macro '__LLONG_MAX'
#define __LLONG_MAX 0x7fffffffffffffffLL /* max value for a long long */
^~~~~~~~~~~~~~~~~~~~
and many instances of:
lib/libc/quad/fixunsdfdi.c:73:17: error: shift count >= width of type [-Werror,-Wshift-count-overflow]
toppart = (x - ONE_HALF) / ONE;
^~~~~~~~
lib/libc/quad/fixunsdfdi.c:45:19: note: expanded from macro 'ONE_HALF'
#define ONE_HALF (ONE_FOURTH * 2.0)
^~~~~~~~~~
lib/libc/quad/fixunsdfdi.c:44:23: note: expanded from macro 'ONE_FOURTH'
#define ONE_FOURTH (1 << (LONG_BITS - 2))
^ ~~~~~~~~~~~~~~~
lib/libc/quad/fixunsdfdi.c:73:29: error: shift count >= width of type [-Werror,-Wshift-count-overflow]
toppart = (x - ONE_HALF) / ONE;
^~~
lib/libc/quad/fixunsdfdi.c:46:15: note: expanded from macro 'ONE'
#define ONE (ONE_FOURTH * 4.0)
^~~~~~~~~~
lib/libc/quad/fixunsdfdi.c:44:23: note: expanded from macro 'ONE_FOURTH'
#define ONE_FOURTH (1 << (LONG_BITS - 2))
^ ~~~~~~~~~~~~~~~
MFC r357861 and r357898:
Improve USB gaming keyboard support.
Add support for decoding pressed keys as a bitmap. The keys in the
bitmap are described in the interface specific HID descriptor. Some
keyboards even have multiple input interfaces, only using the bitmap
method when the event array is full. That typically means when more
than seven keys are pressed simultaneously.
The internals of the USB keyboard driver have been slightly reworked
to keep track of all keys in a single bitmap having 256 bits. This
bitmap is then divided into blocks of 64-bits as an optimisation.
Simplify automatic key repeat logic, because only the last key pressed
can be repeated.
PR: 224592
PR: 233884
Tested by: Alex V. Petrov <alexvpetrov@gmail.com>
Sponsored by: Mellanox Technologies
Kyle Evans [Fri, 21 Feb 2020 04:42:16 +0000 (04:42 +0000)]
MFC r357845: efiloader: don't execute hooks when setting currdev/loaddev
We still need to set the hooks to prevent improper manipulations thereafter
at the loader prompt, but as it is we're actively preventing loaddev from
being set correctly in some circumstances (ZFS in particular) and doing more
work than needed with currdev -- that hook in particular validates it as a
correct device, which we can assume isn't needed in this context.
Kyle Evans [Fri, 21 Feb 2020 04:34:54 +0000 (04:34 +0000)]
MFC r357923-r357924: dtc fixes for /memreserve/ and -I/-O defaults
r357923:
Pull in latest fixes from dtc, up to 0060471
This includes a small battery of /memreserve/ fixes to make sure dtc is
properly writing these regions into the output file and reading them back
out.
As of this update, dtc will now also assume common defaults for -I/-O if
only one is specified; namely, dts for one implies dtb for the other and
vice versa (Requested by: jhibbits, preserves GPL dtc behavior too).
r357924:
dtc: re-apply r353961, r354115
I missed in final review of r357923's diff that these ones hadn't yet been
sent upstream and inadvertently reverted them. =-( Re-apply now.
Alexander Motin [Fri, 21 Feb 2020 04:33:14 +0000 (04:33 +0000)]
MFC r357657: Remove duplicate dbufs accounting.
Since AVL already has embedded element counter, use dn_dbufs_count
only for dbufs not counted there (bonus buffers) and just add them.
This removes two atomics per dbuf life cycle.
According to profiler it reduces time spent by dbuf_destroy() inside
bottlenecked dbuf_evict_thread() from 13.36% to 9.20% of the core.
This counter is used only on illumos, so for FreeBSD it was just a
waste of time.
Kyle Evans [Fri, 21 Feb 2020 04:31:30 +0000 (04:31 +0000)]
MFC r357903: ncurses: correct check for gcc >= 5.0
The hack in question is intended to workaround seemingly bogus #line markers
in cpp output. As far as I can tell, llvm cpp doesn't do this by default, so
there's no reason to add -P.
In our /bin/sh, the main incantation should be placed in a sub-shell in
order to properly pipe the output to fgrep.
The main motivation for this change is admittedly to stop emitting the noise
about clang not being gcc in make -s buildworld
Kyle Evans [Fri, 21 Feb 2020 04:25:10 +0000 (04:25 +0000)]
MFC r357827: lualoader: disable autoboot on high-level interpreter errors
If we hit an error at this level, we are almost certainly not in any kind
of sane state where autoboot can do the right thing. Instead of letting it
try and probably failing, disable autoboot so they immediately get kicked
into a loader prompt for manual remediation/diagnosis.
bhyve: move virtio-net header processing to pci_virtio_net
This patch cleans up the API between the net frontends (e1000,
virtio-net) and the net backends (tap and netmap).
We move the virtio-net header stripping/prepending to the
virtio-net code, where this functionality belongs.
In this way, the netbe_send() and netbe_recv() signatures
can have const struct iov * rather than struct iov *.
Add printf() wrapper to use CR/CRLF terminators depending on whether
stdio is mapped to a tty open in raw mode.
Try to use the wrapper everywhere.
For now we leave the custom DPRINTF/WPRINTF defined by device
models, but we may remove them in the future.
Alexander Motin [Thu, 20 Feb 2020 01:03:44 +0000 (01:03 +0000)]
MFC r357639: Reduce number of atomic_add() calls in aggsum.
Previous code used 4 atomics to do aggsum_flush_bucket() and 2 more to
re-borrow after the flush. But since asc_borrowed and asc_delta are
accessed only while holding asc_lock, it makes no any sense to modify
as_lower_bound and as_upper_bound in multiple steps. Instead of that
the new code uses only 2 atomics in all the cases, one per as_*_bound
variable. I think even that is overkill, simple atomic store and
load could be used here, since all modifications are done under the
as_lock, but there are no such primitives in ZFS code now.
While there, make borrow code consider previous borrow value, so that
on mixed request patterns reduce chance of needing to borrow again if
much larger request follows tiny one that needed borrow.
Also reduce as_numbuckets from uint64_t to u_int. It makes no sense
to use so large division operation on every aggsum_add().
Compared to current version in base:
- great improvements on the Unicode support
- full support for filename completion including quoting
which means we do not need anymore our custom addition)
- Improved readline compatiblity
Upgrading libedit has been a pain in the past, because somehow we never
managed to properly cleanup the tree in lib/libedit and each merge has always
been very painful. After years of fighting give up and refresh a merge from
scrarch properly in contrib.
Note that the switch to this version will be done in another commit.
r352204:
Readd _el_fn_sh_complete for backward compatibility
This function is not needed anymore, it allows old sh binary to continue
to run and avoid breaking backward compatibility.
Note that is now just calls the regular _el_fn_complete which does a proper
job at quoting.
Discussed with: jilles
r352275:
Update libedit to a snapshot from 2019-09-10
This version bring many fixes regarding unicode support
It also adds proper support for filename completion (we do not need our custom
patches anymore)
Improves the libreadline compatibility
Note that the same work was done by Yuichiro Naito in
https://reviews.freebsd.org/D21196 the main difference is in this case we have
reimported libedit in contrib to fix a long standing mess in the previous merges
which prevented a proper update workflow. (discussed long ago with pfg@)
The only difference with upstream libedit is we have added a compatibility shim
for the _elf_fn_sh_complete function which we previously added to support quoting
in filename completion and is not needed anymore.
This was added to continue supported old /bin/sh binaries and not break backward
compatibility (as discussed with jilles@)