MFC r357861 and r357898:
Improve USB gaming keyboard support.
Add support for decoding pressed keys as a bitmap. The keys in the
bitmap are described in the interface specific HID descriptor. Some
keyboards even have multiple input interfaces, only using the bitmap
method when the event array is full. That typically means when more
than seven keys are pressed simultaneously.
The internals of the USB keyboard driver have been slightly reworked
to keep track of all keys in a single bitmap having 256 bits. This
bitmap is then divided into blocks of 64-bits as an optimisation.
Simplify automatic key repeat logic, because only the last key pressed
can be repeated.
PR: 224592
PR: 233884
Tested by: Alex V. Petrov <alexvpetrov@gmail.com>
Sponsored by: Mellanox Technologies
Kyle Evans [Fri, 21 Feb 2020 04:42:16 +0000 (04:42 +0000)]
MFC r357845: efiloader: don't execute hooks when setting currdev/loaddev
We still need to set the hooks to prevent improper manipulations thereafter
at the loader prompt, but as it is we're actively preventing loaddev from
being set correctly in some circumstances (ZFS in particular) and doing more
work than needed with currdev -- that hook in particular validates it as a
correct device, which we can assume isn't needed in this context.
Kyle Evans [Fri, 21 Feb 2020 04:34:54 +0000 (04:34 +0000)]
MFC r357923-r357924: dtc fixes for /memreserve/ and -I/-O defaults
r357923:
Pull in latest fixes from dtc, up to 0060471
This includes a small battery of /memreserve/ fixes to make sure dtc is
properly writing these regions into the output file and reading them back
out.
As of this update, dtc will now also assume common defaults for -I/-O if
only one is specified; namely, dts for one implies dtb for the other and
vice versa (Requested by: jhibbits, preserves GPL dtc behavior too).
r357924:
dtc: re-apply r353961, r354115
I missed in final review of r357923's diff that these ones hadn't yet been
sent upstream and inadvertently reverted them. =-( Re-apply now.
Alexander Motin [Fri, 21 Feb 2020 04:33:14 +0000 (04:33 +0000)]
MFC r357657: Remove duplicate dbufs accounting.
Since AVL already has embedded element counter, use dn_dbufs_count
only for dbufs not counted there (bonus buffers) and just add them.
This removes two atomics per dbuf life cycle.
According to profiler it reduces time spent by dbuf_destroy() inside
bottlenecked dbuf_evict_thread() from 13.36% to 9.20% of the core.
This counter is used only on illumos, so for FreeBSD it was just a
waste of time.
Kyle Evans [Fri, 21 Feb 2020 04:31:30 +0000 (04:31 +0000)]
MFC r357903: ncurses: correct check for gcc >= 5.0
The hack in question is intended to workaround seemingly bogus #line markers
in cpp output. As far as I can tell, llvm cpp doesn't do this by default, so
there's no reason to add -P.
In our /bin/sh, the main incantation should be placed in a sub-shell in
order to properly pipe the output to fgrep.
The main motivation for this change is admittedly to stop emitting the noise
about clang not being gcc in make -s buildworld
Kyle Evans [Fri, 21 Feb 2020 04:25:10 +0000 (04:25 +0000)]
MFC r357827: lualoader: disable autoboot on high-level interpreter errors
If we hit an error at this level, we are almost certainly not in any kind
of sane state where autoboot can do the right thing. Instead of letting it
try and probably failing, disable autoboot so they immediately get kicked
into a loader prompt for manual remediation/diagnosis.
bhyve: move virtio-net header processing to pci_virtio_net
This patch cleans up the API between the net frontends (e1000,
virtio-net) and the net backends (tap and netmap).
We move the virtio-net header stripping/prepending to the
virtio-net code, where this functionality belongs.
In this way, the netbe_send() and netbe_recv() signatures
can have const struct iov * rather than struct iov *.
Add printf() wrapper to use CR/CRLF terminators depending on whether
stdio is mapped to a tty open in raw mode.
Try to use the wrapper everywhere.
For now we leave the custom DPRINTF/WPRINTF defined by device
models, but we may remove them in the future.
Alexander Motin [Thu, 20 Feb 2020 01:03:44 +0000 (01:03 +0000)]
MFC r357639: Reduce number of atomic_add() calls in aggsum.
Previous code used 4 atomics to do aggsum_flush_bucket() and 2 more to
re-borrow after the flush. But since asc_borrowed and asc_delta are
accessed only while holding asc_lock, it makes no any sense to modify
as_lower_bound and as_upper_bound in multiple steps. Instead of that
the new code uses only 2 atomics in all the cases, one per as_*_bound
variable. I think even that is overkill, simple atomic store and
load could be used here, since all modifications are done under the
as_lock, but there are no such primitives in ZFS code now.
While there, make borrow code consider previous borrow value, so that
on mixed request patterns reduce chance of needing to borrow again if
much larger request follows tiny one that needed borrow.
Also reduce as_numbuckets from uint64_t to u_int. It makes no sense
to use so large division operation on every aggsum_add().
Compared to current version in base:
- great improvements on the Unicode support
- full support for filename completion including quoting
which means we do not need anymore our custom addition)
- Improved readline compatiblity
Upgrading libedit has been a pain in the past, because somehow we never
managed to properly cleanup the tree in lib/libedit and each merge has always
been very painful. After years of fighting give up and refresh a merge from
scrarch properly in contrib.
Note that the switch to this version will be done in another commit.
r352204:
Readd _el_fn_sh_complete for backward compatibility
This function is not needed anymore, it allows old sh binary to continue
to run and avoid breaking backward compatibility.
Note that is now just calls the regular _el_fn_complete which does a proper
job at quoting.
Discussed with: jilles
r352275:
Update libedit to a snapshot from 2019-09-10
This version bring many fixes regarding unicode support
It also adds proper support for filename completion (we do not need our custom
patches anymore)
Improves the libreadline compatibility
Note that the same work was done by Yuichiro Naito in
https://reviews.freebsd.org/D21196 the main difference is in this case we have
reimported libedit in contrib to fix a long standing mess in the previous merges
which prevented a proper update workflow. (discussed long ago with pfg@)
The only difference with upstream libedit is we have added a compatibility shim
for the _elf_fn_sh_complete function which we previously added to support quoting
in filename completion and is not needed anymore.
This was added to continue supported old /bin/sh binaries and not break backward
compatibility (as discussed with jilles@)
r352247:
Remove useless extra definition of libedit flags
Note that all the line editor part is done in the libntp
r352248:
Get the readline header from the installed header instead of the from the source
location.
With newer import of libedit, the path to be able to access readline/readline.h
will also include header which name will conflict with some expected by ntp in
another path and end up breaking the build.
Those scripts are without copyright and license assignement since their creation
After grabbing information from The various authors and contributors assign
proper license header and copyrights.
This has been reported by yuripv in his work on integrating those in Illumos!
The ports tree now accepts /usr/local/share/man as a directory for manpage
and will slowly transition from /usr/local/man to it. To reflect this remove
the documentation of the manpages being an exception in the layout of /usr/local
Reported by: Dan Nelson <dnelson_1901@yahoo.com> (via IRC)
Martin Matuska [Wed, 19 Feb 2020 01:46:43 +0000 (01:46 +0000)]
MFC r356212,r356366,r356416,r357785
Update libarchive to version 3.4.2
Relevant vendor changes (r356212):
Issue #351: Refactor and implement private state logic for write filters
PR #1252: RAR5 reader - verify window size for solid files (OSS-Fuzz 15482)
PR #1255: zip writer - don't append unused NUL for directories
PR #1260: Fix sparse file offset overflow on 32-bit systems
PR #1263: UNICODE filename support for reading lha/lzh format
Issue #1276: Bugfix and optimize archive_wstring_append_from_mbs()
PR #1288: Add the "xattrhdr" option to pax write options
PR #1295: 7z reader - fix reading archives with digests in PackInfo
PR #1296: RAR5 reader - verify window size for multivolume archives
PR #1297: ZIP reader - support LZMA_STREAM_END marker in 'lzma alone' files
Issue #1298: Fix a heap-buffer-overflow in archive_string_append_from_wcs()
OSS-Fuzz 19360, 19362: LHA reader - plug two memory leaks on error
Fix possible off-by-one when dealing with readlink(2)
Relevant vendor changes (r356366):
Issue #1302: Plug memory leak on failure of archive_write_client_open()
Relevant vendor changes (r356416):
Issue #1302: Re-do fix for archive_write_client_open()
Relevant vendor changes (r357785):
PR #1289: atomic extraction support (bsdtar -x --safe-writes)
PR #1308: big endian fix for UTF16 support in LHA reader
PR #1326: reject RAR5 files that declare invalid header flags
Issue #987: fix support 7z archive entries with Delta filter
Issue #1317: fix compression output buffer handling in XAR writer
Issue #1319: fix uname or gname longer than 32 characters in pax writer
Issue #1325: fix use after free when archiving hardlinks in ISO9660 or XAR
Use localtime_r() and gmtime_r() instead of localtime() and gmtime()
Brooks Davis [Tue, 18 Feb 2020 21:36:54 +0000 (21:36 +0000)]
MFC r357788:
Mark hme(4) as deprecated.
It was saved from the initial purge of drivers in fcp-101 due to being
the onboard Ethernet device on a number of sparc64 machines. Now that
sparc64 is gone, it serves little purpose (PCI cards exist, but are rare
and are unlikely to have been deployed outside Sun systems).
Dimitry Andric [Tue, 18 Feb 2020 18:17:03 +0000 (18:17 +0000)]
MFC r357976:
Merge r357970 from the clang1000-import branch:
Fix the following -Werror warning from clang 10.0.0 in hptmv(4):
sys/dev/hptmv/ioctl.c:240:4: error: misleading indentation; statement is not part of the previous 'if' [-Werror,-Wmisleading-indentation]
_vbus_p=pArray->pVBus;
^
sys/dev/hptmv/ioctl.c:237:10: note: previous statement is here
if(!mIsArray(pArray))
^
This is because the return statement after the if statement was not
indented. (Note that this file has been idented assuming 4-space tabs.)
Dimitry Andric [Tue, 18 Feb 2020 18:15:33 +0000 (18:15 +0000)]
MFC r357873:
Merge r357864 from the clang1000-import branch:
Fix the following -Werror warning from clang 10.0.0 in sconfig:
sbin/sconfig/sconfig.c:909:5: error: misleading indentation; statement is not part of the previous 'else' [-Werror,-Wmisleading-indentation]
exit (-1);
^
sbin/sconfig/sconfig.c:907:6: note: previous statement is here
} else
^
The intent was to group the exit() call with the previous fprintf()
call.
Alexander Motin [Tue, 18 Feb 2020 01:21:56 +0000 (01:21 +0000)]
MFC r357502: Few microoptimizations to dbuf layer.
Move db_link into the same cache line as db_blkid and db_level.
It allows significantly reduce avl_add() time in dbuf_create() on
systems with large RAM and huge number of dbufs per dnode.
Avoid few accesses to dbuf_caches[].size, which is highly congested
under high IOPS and never stays in cache for a long time. Use local
value we are receiving from zfs_refcount_add_many() any way.
Remove cache_size_bytes_max bump from dbuf_evict_one(). I don't see
a point to do it on dbuf eviction after we done it on insertion in
dbuf_rele_and_unlock().
Rick Macklem [Mon, 17 Feb 2020 19:32:54 +0000 (19:32 +0000)]
MFC: r357149
Fix a crash in the NFSv4 server.
The PR reported a crash that occurred when a file was removed while
client(s) were actively doing lock operations on it.
Since nfsvno_getvp() will return NULL when the file does not exist,
the bug was obvious and easy to fix via this patch. It is a little
surprising that this wasn't found sooner, but I guess the above
case rarely occurs.
Colin Percival [Fri, 14 Feb 2020 02:31:20 +0000 (02:31 +0000)]
MFC r353022:
Switch EC2 AMIs from using the dual-dhclient script to using the new
dual-dhclient-daemon daemon. This makes it possible to stop/restart
the dhclients.
Kyle Evans [Fri, 14 Feb 2020 02:27:01 +0000 (02:27 +0000)]
MFC r357756: backup-passwd: mask out all passwords in the diff
The previous expression borked if a username had a plus or hyphen in it.
This is needlessly restrictive- at leSt a hyphen in the middle is valid.
Instead of playing this game, let's just assume the username can't contain a
colon and mask out the second field.
Alan Somers [Thu, 13 Feb 2020 20:49:45 +0000 (20:49 +0000)]
MFC r357235:
Speed up "zpool import" in the presence of many zvols
By default, zpools may not be backed by zvols (that can be changed with the
"vfs.zfs.vol.recursive" sysctl). When that sysctl is set to 0, the kernel
does not attempt to read zvols when looking for vdevs. But the zpool command
still does. This change brings the zpool command into line with the kernel's
behavior. It speeds "zpool import" when an already imported pool has many
zvols, or a zvol with many snapshots.
PR: 241083
Reported by: Martin Birgmeier <d8zNeCFG@aon.at>
Reviewed by: mav, Ryan Moeller <ryan@freqlabs.com>
Sponsored by: Axcient
Differential Revision: https://reviews.freebsd.org/D22077
Alan Somers [Thu, 13 Feb 2020 20:45:32 +0000 (20:45 +0000)]
MFC r356809:
setextattr: Increase stdin buffer size to 4096
Extended attribute values can potentially be quite large. One test for ZFS
is supposed to set a 200MB xattr. However, the buffer size for reading
values from stdin with setextattr -i is so small that the test times out
waiting for tiny chunks of data to be buffered and appended to an sbuf.
Increasing the buffer size should help alleviate some of the burden of
reallocating larger sbufs when writing large extended attributes.
Submitted by: Ryan Moeller <ryan@freqlabs.com>
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D23211
Alan Somers [Thu, 13 Feb 2020 20:40:36 +0000 (20:40 +0000)]
MFC r355430:
ses: sanitize illegal strings in SES element descriptors
The SES4r3 standard requires that element descriptors may only contain ASCII
characters in the range 0x20 to 0x7e. Some SuperMicro expanders violate
that rule. This patch adds a sanity check to ses(4). Descriptors in
violation will be replaced by "<invalid>".
This patch fixes "sesutil --libxo xml" on such systems. Previously it would
generate non-well-formed XML output.
Alan Somers [Thu, 13 Feb 2020 20:32:05 +0000 (20:32 +0000)]
MFC r355431:
gmultipath: add ATF tests
Add ATF tests for most gmultipath operations. Add some dtrace probes too,
primarily for configuration changes that happen in response to provider
errors.
Kyle Evans [Thu, 13 Feb 2020 03:13:29 +0000 (03:13 +0000)]
MFC r357284, r357419: stdio unlocked
r357284:
stdio: provide _unlocked variants of fflush, fputc, fputs, fread, fwrite
fflush_unlocked is currently desired in ports by sysutils/metalog, and
redefined as the locked fflush.
fputc_unlocked, fputs_unlocked, fread_unlocked, and fwrite_unlocked are
currently desired in ports by devel/elfutils, and redefined as the locked
fputs, fread, and fwrite respectively.
r357419:
libc: provide fputc_unlocked
Among the same justification as the other stdio _unlocked; in addition to an
inline version in <stdio.h>, we must provide a function in libc as well for
the functionality. This fixes the lang/gcc* builds, which want to use the
symbol from libc.
Kyle Evans [Wed, 12 Feb 2020 02:17:20 +0000 (02:17 +0000)]
MFC r357665: geli taste: allow GELIBOOT tagged providers as well
Currently the installer will tag geliboot partitions with both BOOT and
GELIBOOT; the former allows the kernel to taste it at boot, while the latter
is what loaders keys off of.
However, it seems reasonable to assume that if a provider's been tagged with
GELIBOOT that the kernel should also take that as a hint to taste/attach at
boot. This would allow us to stop tagging GELIBOOT partitions with BOOT in
bsdinstall, but I'm not sure that there's a compelling reason to do so any
time soon.
O_SEARCH is defined by POSIX [0] to open a directory for searching, skipping
permissions checks on the directory itself after the initial open(). This is
close to the semantics we've historically applied for O_EXEC on a directory,
which is UB according to POSIX. Conveniently, O_SEARCH on a file is also
explicitly undefined behavior according to POSIX, so O_EXEC would be a fine
choice. The spec goes on to state that O_SEARCH and O_EXEC need not be
distinct values, but they're not defined to be the same value.
This was pointed out as an incompatibility with other systems that had made
its way into libarchive, which had assumed that O_EXEC was an alias for
O_SEARCH.
This defines compatibility O_SEARCH/FSEARCH (equivalent to O_EXEC and FEXEC
respectively) and expands our UB for O_EXEC on a directory. O_EXEC on a
directory is checked in vn_open_vnode already, so for completeness we add a
NOEXECCHECK when O_SEARCH has been specified on the top-level fd and do not
re-check that when descending in namei.
r357461:
namei: preserve errors from fget_cap_locked
Most notably, we want to make sure we don't clobber any capabilities-related
errors. This is a regression from r357412 (O_SEARCH) that was picked up by
the capsicum tests.
r357580:
O_SEARCH test: drop O_SEARCH|O_RDWR local diff
In FreeBSD's O_SEARCH implementation, O_SEARCH in conjunction with O_RDWR or
O_WRONLY is explicitly rejected. In this case, O_RDWR was not necessary
anyways as the file will get created with or without it.
This was submitted upstream as misc/54940 and committed in rev 1.8 of the
file.
r357584:
Record-only MFV of r357583: netbsd-tests: import upstreamed changes
The changes in question originated in FreeBSD/head; no further action is
required.
r357636:
MFV r357635: imnport v1.9 of the O_SEARCH tests
The RCSID data was wrong, so this is effectively a record-only merge
with correction of said data. No further changes should be needed in this
area, as we've now upstreamed our local changes to this specific test.
r357671:
O_SEARCH test: mark revokex an expected fail on NFS
The revokex test does not work when the scratch directory is created on NFS.
Given the nature of NFS, it likely can never work without looking like a
security hole since O_SEARCH would rely on the server knowing that the
directory did have +x at the time of open and that it's OK for it to have
been revoked based on POSIX specification for O_SEARCH.
This does mean that O_SEARCH is only partially functional on NFS in general,
but I suspect the execute bit getting revoked in the process is likely not
common.
r357688:
MFV r357687: Import NFS fix for O_SEARCH tests
The version that ended upstream was ultimately slightly different than the
version committed here; notably, statvfs() is used but it's redefined
appropriately to statfs() on FreeBSD since we don't provide the fstypename
for the former interface.
Kyle Evans [Sun, 9 Feb 2020 19:20:20 +0000 (19:20 +0000)]
MFC r357572: wc(1): account for possibility of file == NULL
file could reasonably be NULL here if we we're using stdin. Albeit less
likely in normal usage, one could actually hit either of these warnings on
stdin.
Alexander Motin [Sat, 8 Feb 2020 21:55:56 +0000 (21:55 +0000)]
MFC r353678 (by avg):
provide a way to assign taskqueue threads to a kernel process
This can be used to group all threads belonging to a single logical
entity under a common kernel process.
I am planning to use the new interface for ZFS threads.
Kyle Evans [Fri, 7 Feb 2020 21:55:45 +0000 (21:55 +0000)]
MFC r357509-r357510: small psm cleanup
r357509:
psm: release resources on attach failure
In exactly 1/3 cases we'll release resources on failure; touch up the other
two to do so as well.
r357510:
psm: use make_dev_s instead of make_dev
This most importantly reduces duplication, but it also removes any potential
race with usage of dev->si_drv1 since it's now set prior to the device being
constructed enough to be accessible.
Kyle Evans [Fri, 7 Feb 2020 19:00:43 +0000 (19:00 +0000)]
MFC r355248: tty: implement TIOCNOTTY
Generally, it's preferred that an application fork/setsid if it doesn't want
to keep its controlling TTY, but it could be that a debugger is trying to
steal it instead -- so it would hook in, drop the controlling TTY, then do
some magic to set things up again. In this case, TIOCNOTTY is quite handy
and still respected by at least OpenBSD, NetBSD, and Linux as far as I can
tell.
I've dropped the note about obsoletion, as I intend to support TIOCNOTTY as
long as it doesn't impose a major burden.
Kyle Evans [Thu, 6 Feb 2020 18:40:37 +0000 (18:40 +0000)]
MFC r357195, r357556: O_SEARCH test fixes
r357195:
netbsd-tests: libc: use correct modes in O_SEARCH tests
The current code clearly intended for these to be octal based on the values
used, but the octal prefix was forgotten. Add it now for correctness, but
note that we don't currently execute these tests.
This has been submitted upstream as misc/54902, so I've omitted the standard
FreeBSD markers that we tend to put into netbsd-tests for upstream-candidate
identification.
r357556:
O_SEARCH tests: plug trivial fd leak
Coverity correctly reports this as a resource leak. It's an admittedly minor
one, but plug it anyways.
Kyle Evans [Thu, 6 Feb 2020 18:37:38 +0000 (18:37 +0000)]
MFC r357193: caroot: blacklisted: automatically pick up *.pem in the tree
This kind of automagica got picked up in trusted/ prior to the initial
commit, but never got applied over in blacklisted. Ideally no one will be
using blacklisted/ to store arbitrary certs that they don't intend to
blacklist, so we should just install anything that's in here rather than
force consumer to first copy cert into place and then modify the file
listing in the Makefile.
Wise man once say: "it is better to restrict too much, than not enough.
sometimes."
Kyle Evans [Wed, 5 Feb 2020 21:30:31 +0000 (21:30 +0000)]
MFC r357410-r357411: Avoid duplicating VEXEC checks in VOP_CACHEDLOOKUP
r357410: pseudofs: don't do VEXEC check in VOP_CACHEDLOOKUP
VOP_CACHEDLOOKUP should assume that the appropriate VEXEC check has been
done in the caller (vfs_cache_lookup), so it does not belong here.
r357411: zfs: light refactor to indicate cachedlookup in zfs_lookup
If we come from VOP_CACHEDLOOKUP, we must skip the VEXEC check as it will
have been done in the caller (vfs_cache_lookup). This is a part of D23247,
which may skip the earlier VEXEC check as well if the root fd was opened
with O_SEARCH.
This one required slightly more work as zfs_lookup may also be called
indirectly as VOP_LOOKUP or a couple of other places where we must do the
check.
MFCr356926:
Even when the MK_CASPER is set to "no" we still want to install man pages
and the headers. If the user decides to install the system without Casper
support, then the Casper functions are mocked, but they still exist in
the system.