Warner Losh [Sat, 31 Jul 2021 22:17:44 +0000 (16:17 -0600)]
awk: use awkgram.tab.h consistently
yacc makes awkgram.h. However, one true awk includes awkgram.tab.h, so
we link to for the builds. Make sure that we consistently link to it.
Also, restore the awkgram.tab.h dependency to maketab. It should not
have been deleted, despite apparently making meta build on stable/12
work. The important missing arc was proctab.c's dependence on
awkgram.tab.h.
Warner Losh [Sat, 31 Jul 2021 21:41:29 +0000 (15:41 -0600)]
awk: Fix dependencies
proctab.c is generated from awktab.h, so needs to depend on it.
maketab does not depend on awktab.h, and gets the maketab.c dependency
automatically, so remove them both.
Normally, these don't matter. However, for a meta build, they can cause
us to build maketab twice (once host, once for target) resulting in a
binary that can't run on the host due to proctab.c racing maketab in
parallel legs. In stable/12, this was a reliably lost race, while in
main I've been unable to trigger the race at all (maybe due to dirdep
changes making main more robust).
amd64: do not assume that kernel is loaded at 2M physical
Allow any 2M aligned contiguous location below 4G for the staging
area location. It should still be mapped by loader at KERNBASE.
The assumption kernel makes about loader->kernel handoff with regard to
the MMU programming are explicitly listed at the beginning of hammer_time(),
where kernphys is calculated. Now kernphys is the variable instead of
symbol designating the physical address.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D31121
Warner Losh [Sat, 31 Jul 2021 05:31:00 +0000 (23:31 -0600)]
awk: Document deprecated behavior of hex constants and locales.
FreeBSD will convert "0x12" from hex and print it as 18. Other awks will
convert it to 0. This extension has been removed upstream, and will be
removed in FreeBSD 14.0.
FreeBSD used to set the locale on startup, and make the ranges use that
locale. This lead to weird results like "[A-Z]" matching lower case
characters in some locales. This bug has been fixed.
Warner Losh [Sat, 31 Jul 2021 05:19:58 +0000 (23:19 -0600)]
awk: Flag -Ft as deprecated behavior
Upstream is poised to deprecate the -Ft wart in one true awk. None of
the other awks do this, and the gawk maintainer says that he's had no
requests for it in gawk in 30 years maintaining it. github can find a
few instances of it in the wild. As such, warn that it's deprecated and
will go away in the future.
Warner Losh [Fri, 30 Jul 2021 23:11:43 +0000 (17:11 -0600)]
clock_gettime: Add Linux aliases for CLOCK_*
Linux standardized what we call CLOCK_{REALTIME,MONOTONIC}_FAST as
CLOCK_{REALTIME,MONOTONIC}_COARSE. In addition, Linux spells
CLOCK_UPTIME as CLOCK_BOOTTIME.
Add aliases to time.h and document these new aliases in
clock_gettime(2).
Warner Losh [Fri, 30 Jul 2021 23:10:56 +0000 (17:10 -0600)]
time.h: reduce CLOCK_ namespace pollution, move to _clock_id.h
Attempt to comply with the strict namespace pollution requirements of
_POSIX_C_SOURCE. Add guards to limit visitbility of CLOCK_ and TIMER_
defines as appropriate. Only define the CLOCK_ variables relevant to the
specific standards. Move all the sharing to sys/_clock_id.h and make
time.h and sys/time.h both include that rather than copy due to the
now large number of clocks and compat defines.
Please note: The old time.h previously used these newer dates:
CLOCK_REALTIME 199506
CLOCK_MONOTONIC 200112
CLOCK_THREAD_CPUTIME_ID 200112
CLOCK_PROCESS_CPUTIME_ID 200112
but glibc defines all of these for 199309. glibc uses this date for all
these values, however, only CLOCK_REALTIME was in IEEE 1003.1b. Add a
comment about this to document it. A large number of programs and
libraries assume that these will be defined for _POSIX_C_SOURCE =
199309.
In addition, leak CLOCK_UPTIME_FAST for the pocl package until it can be
updated to use a simple CLOCK_MONOTONIC.
While nvlists are very useful in maximising flexibility for future
extensions their performance is simply unacceptably bad for the
getstates feature, where we can easily want to export a million states
or more.
The DIOCGETSTATESNV call has been MFCd, but has not hit a release on any
branch, so we can still remove it everywhere.
Alexander Motin [Fri, 30 Jul 2021 03:39:04 +0000 (23:39 -0400)]
ipmi(4): Add more watchdog error checks.
Add request submission status checks before checking req->ir_compcode,
otherwise it may be zero just because of initialization.
Add checks for req->ir_compcode errors in ipmi_reset_watchdog() and
ipmi_set_watchdog(). In first case explicitly check for 0x80, which
means timer was not previously set, that I found happening after BMC
cold reset. This change makes watchdog timer to recover instead of
permanently ignoring reset errors after BMC reset or upgraded.
Alexander Motin [Fri, 30 Jul 2021 03:16:22 +0000 (23:16 -0400)]
coretemp(4): Switch to smp_rendezvous_cpus().
Use of smp_rendezvous_cpus() instead of sched_bind() allows to not
block indefinitely if target CPU is running some thread with higher
priority, while all we need is single rdmsr/wrmsr instruction call.
I guess it should also be much cheaper than full thread migration.
Mark Johnston [Mon, 19 Jul 2021 20:09:42 +0000 (16:09 -0400)]
Simplify kernel sanitizer interceptors
KASAN and KCSAN implement interceptors for various primitive operations
that are not instrumented by the compiler. KMSAN requires them as well.
Rather than adding new cases for each sanitizer which requires
interceptors, implement the following protocol:
- When interceptor definitions are required, define
SAN_NEEDS_INTERCEPTORS and SANITIZER_INTERCEPTOR_PREFIX.
- In headers that declare functions which need to be intercepted by a
sanitizer runtime, use SANITIZER_INTERCEPTOR_PREFIX to provide
declarations.
- When SAN_RUNTIME is defined, do not redefine the names of intercepted
functions. This is typically the case in files which implement
sanitizer runtimes but is also needed in, for example, files which
define ifunc selectors for intercepted operations.
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
John Baldwin [Fri, 30 Jul 2021 00:09:23 +0000 (17:09 -0700)]
geom_vfs: Pre-allocate event for g_vfs_destroy.
When an active g_vfs is orphaned due to an underlying disk going away
the destroy is deferred until the filesystem is unmounted in
g_vfs_done(). However, g_vfs_done() is invoked from a non-sleepable
context and cannot use M_WAITOK to allocate the event. Instead,
allocate the event in g_vfs_orphan() and save it in the softc to be
retrieved by the last call to g_vfs_done().
John Baldwin [Thu, 29 Jul 2021 23:34:46 +0000 (16:34 -0700)]
cxgbei: Wait for socket to close in icl_cxgbei_conn_close.
This ensures the TOE has finished processing any in-flight received
data before returning to the caller. The caller assumes it is safe to
free any open tasks or transfers (and associated buffers) after this
function returns.
Previously, data placed directly via DDP could be written to buffers
after the caller had freed the buffers.
Clean up orphaned indirdep dependency structures after disk failure.
During forcible unmount after a disk failure there is a bug that
causes one or more indirdep dependency structures to fail to be
deallocated. Until we manage to track down why they fail to get
cleaned up, this code tracks them down and eliminates them so that
the unmount can succeed.
Reported by: Peter Holm
Help from: kib
Reviewed by: Chuck Silvers
Tested by: Peter Holm
MFC after: 7 days
Sponsored by: Netflix
Diagnotic improvement to soft dependency structure management.
The soft updates diagnotic code keeps a list for each type of soft
update dependency. When a new block is allocated for a file it is
initially tracked by a "newblk" dependency. The "newblk" dependency
eventually becomes either an "allocdirect" dependency or an "indiralloc"
dependency. The diagnotic code failed to move the "newblk" from the list
of "newblk"s to its new type list.
No functional change intended.
Reviewed by: Chuck Silvers (as part of a larger change)
Tested by: Peter Holm (as part of a larger change)
Sponsored by: Netflix
amd64: stop doing special allocation for the AP startup trampoline
There is no reason now why do we need to allocate trampoline page very
early in the boot process. The only requirement for the page is that
it is below 1M to be usable by the real mode during init. This can be
handled by vm_alloc_contig() when we do the startup.
Also assert that startup trampoline fits into single page. In principle
we can do multi-page allocation if needed, but it is not.
Move the alloc_ap_trampoline() function and the boot_address variable to
i386/mp_machdep.c. Keep existing mechanism of early alloc on i386.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D31343
Mark Johnston [Thu, 29 Jul 2021 14:22:37 +0000 (10:22 -0400)]
amd64: Set GS.base before calling init_secondary() on APs
KMSAN instrumentation requires thread-local storage to track
initialization state for function parameters and return values. This
buffer is accessed as part of each function prologue. It is provided by
the KMSAN runtime, which looks up a pointer in the current thread's
structure.
When KMSAN is configured, init_secondary() is instrumented, but this
means that GS.base must be initialized first, otherwise the runtime
cannot safely access curthread. Work around this by loading GS.base
before calling init_secondary(), so that the runtime can at least check
curthread == NULL and return a pointer to some dummy storage. Note that
init_secondary() still must reload GS.base after calling lgdt(), which
loads a selector into %gs, which in turn clears the base register.
Reviewed by: kib
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31336
Mark Johnston [Thu, 29 Jul 2021 13:46:25 +0000 (09:46 -0400)]
link_elf_obj: Invoke fini callbacks
This is required for KASAN: when a module is unloaded, poisoned regions
(e.g., pad areas between global variables) are left as such, so if they
are reused as KLDs are loaded, false positives can arise.
Reported by: pho, Jenkins
Reviewed by: kib
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31339
FUTEX_LOCK_PI2 was added to support clock selection as FUTEX_LOCK_PI uses a
CLOCK_REALTIME based absolute value since it was implemented, but it does not
require that the FUTEX_CLOCK_REALTIME bit is set, because that was introduced
later.
Linux futex documentation explicitly states that EINVAL is returned if
the futex is not 4-byte aligned. Check futex alignment as a Linux do
and return EINVAL.
The bitset is a Linux emulation layer extension. This 32-bit mask, in which at
least one bit must be set, is used to select which threads should be woken up.
The bitset is stored in the umtx_q structure, which is used to enqueue the waiter
into the umtx waitqueue. Put the bitset into the hole, that appeared on LP64 due
to data alignment, to prevent the growth of the struct umtx_q.
Wojciech Macek [Thu, 29 Jul 2021 09:02:43 +0000 (11:02 +0200)]
Fix mac_veriexec version mismatch
mac_veriexec sets its version to 1, but the mac_veriexec_shaX modules which depend on it expect MAC_VERIEXEC_VERSION = 2.
Be consistent and use MAC_VERIEXEC_VERSION everywhere.
This unbreaks loading of mac_veriexec modules at boot time.
virtio: enable VTNET_LEGACY_TX when ALTQ is enabled.
ALTQ only works on network drivers which use if_start (rather than
if_transmit). vtnet uses if_start if built with VTNET_LEGACY_TX. Default
to that the kernel is built with ALTQ enabled, to reduce user surprise.
gcc failed as it didn't inlined the builtins and generates calls to
the libgcc, ld can't find libgcc as cross-toolchain libgcc is not installed.
To avoid this add internal vDSO ffs functions without optimized builtins.
The left side of the MIN() expression is the (signed) result of pointer
subtraction (ptrdiff_t). The right hand side is the also the (signed)
result of pointer subtraction, additionally subtracting the element size
('es'), which is unsigned size_t. This coerces the right-hand
expression into an unsigned value. MIN(signed, unsigned) triggers
-Wsign-compare.
Sorting elements of size greater than SSIZE_MAX is nonsensical, so we
can instead treat the element size as ssize_t, leaving the right-hand
result the same signedness as the left.
makesyscalls was rewritten in Lua and introduced in d3276301ab. In the
time since, no objections have risen and a warning was introduced long
ago on invocation of makesyscalls.sh that it would be removed before
FreeBSD 13. Belatedly follow through on that.
Alexander Motin [Thu, 29 Jul 2021 01:18:50 +0000 (21:18 -0400)]
Refactor/optimize cpu_search_*().
Remove cpu_search_both(), unused for many years. Without it there is
less sense for the trick of compiling common cpu_search() into separate
cpu_search_lowest() and cpu_search_highest(), so split them completely,
making code more readable. While there, split iteration over children
groups and CPUs, complicating code for very small deduplication.
Stop passing cpuset_t arguments by value and avoid some manipulations.
Since MAXCPU bump from 64 to 256, what was a single register turned
into 32-byte memory array, requiring memory allocation and accesses.
Splitting struct cpu_search into parameter and result parts allows to
even more reduce stack usage, since the first can be passed through
on recursion.
Remove CPU_FFS() from the hot paths, precalculating first and last CPU
for each CPU group in advance during initialization. Again, it was
not a problem for 64 CPUs before, but for 256 FFS needs much more code.
With these changes on 80-thread system doing ~260K uncached ZFS reads
per second I observe ~30% reduction of time spent in cpu_search_*().
debugnet: Fix false-positive assertions for dp_state
debugnet_handle_arp:
An assertion is present to ensure the pcb is only modified when the state is
DN_STATE_INIT. Because debugnet_arp_gw() is asynchronous it is possible for
ARP replies to come in after the gateway address is known and the state
already changed.
debugnet_handle_ip:
Similarly it is possible for packets to come in, from the expected
server, during the gateway mac discovery phase. This can happen from
testing disconnects / reconnects in quick succession. This later
causes some acks to be sent back but hit an assertion because the
state is wrong.
Rick Macklem [Wed, 28 Jul 2021 22:48:27 +0000 (15:48 -0700)]
nfscl: Cache an open stateid for the "oneopenown" mount option
For NFSv4.1/4.2, if the "oneopenown" mount option is used,
there is, at most, only one open stateid for each NFS vnode.
When an open stateid for a file is acquired, set a pointer to
the open structure in the NFS vnode. This pointer can be used to
acquire the open stateid without searching the open linked list
when the following is true:
- No delegations have been issued for the file. Since delegations
can outlive an NFS vnode for a file, use the global
NFSMNTP_DELEGISSUED flag on the mount to determine this.
- No lock stateid has been issued for the file. To determine
this, a new NFS vnode flag called NMIGHTBELOCKED is set when a lock
stateid is issued, which can then be tested.
When this open structure pointer can be used, it avoids the need to
acquire the NFSCLSTATELOCK() and searching the open structure list for
an open. The NFSCLSTATELOCK() can be highly contended when there are
a lot of opens issued for the NFSv4.1/4.2 mount.
This patch only affects NFSv4.1/4.2 mounts when the "oneopenown"
mount option is used.
Rick Macklem [Wed, 28 Jul 2021 22:23:05 +0000 (15:23 -0700)]
nfscl: Set correct lockowner for "oneopenown" mount option
For NFSv4.1/4.2, the client may use either an open, lock or
delegation stateid as the stateid argument for an I/O operation.
RFC 5661 defines an order of preference of delegation, then lock
and finally open stateid for the argument, although NFSv4.1/4.2
servers are expected to handle any stateid type.
For the "oneopenown" mount option, the lock owner was not being
correctly generated and, as such, the I/O operation would use an
open stateid, even when a lock stateid existed. Although this
did not and should not affect an NFSv4.1/4.2 server's behaviour,
this patch makes the behaviour for "oneopenown" the same as when
the mount option is not specified.
Found during inspection of packet captures. No failure during
testing against NFSv4.1/4.2 servers of the unpatched code occurred.
Ed Maste [Wed, 28 Jul 2021 20:02:49 +0000 (16:02 -0400)]
pkgbase: improve pkg --version parsing
In some cases `pkg --version` might produce unexpected or additional
output. Use a regex /^[0-9.]+$/ to match only the line containing the
version number.
Reported by: Michael Butler on freebsd-current@
Fixes: 4e224e4be7c3 ("pkgbase: accommodate pkg < 1.17")
Sponsored by: The FreeBSD Foundation
Alexander Motin [Wed, 28 Jul 2021 20:15:43 +0000 (16:15 -0400)]
Do not expose to scheduler caches of single CPU.
Before this change my dual-Xeon(R) Gold 6242R always reported 3 levels
or topology (root, package/L3 and core/L2). But with SMT disabled
core/L2 matches thread, so additional topology level only causes more
traversal work. With this change SMT case is reported same as before,
while non-SMT is reported with only 2 much more simple levels.
compilert-rt: build out-of-line LSE atomics helpers for aarch64
Both clang >= 12 and gcc >= 10.1 now default to -moutline-atomics for
aarch64. This requires a bunch of helper functions in libcompiler_rt.a,
to avoid link errors like "undefined symbol: __aarch64_ldadd8_acq_rel".
(Note: of course you can use -mno-outline-atomics as a workaround too,
but this would negate the potential performance benefit of the faster
LSE instructions.)
Bump __FreeBSD_version so ports maintainers can easily detect this.
if_bridge used to only allow MTU changes if the new MTU matched that of
all member interfaces. This doesn't really make much sense, in that we
really shouldn't be allowed to change the MTU of bridge member in the
first place.
Instead we now change the MTU of all member interfaces. If one fails we
revert all interfaces back to the original MTU.
We do not address the issue where bridge member interface MTUs can be
changed here.
John Hood [Wed, 28 Jul 2021 19:43:02 +0000 (13:43 -0600)]
loader: support.4th resets the read buffer incorrectly
Large nextboot.conf files (over 80 bytes) are not read correctly by the
Forth loader, causing file parsing to abort, and nextboot configuration
fails to apply.
Simple repro:
nextboot -e foo=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
shutdown -r now
That will cause the bug to cause a parse failure but shouldn't otherwise
affect the boot. Depending on your loader configuration, you may also
have to set beastie_disable and/or reduce the number of modules loaded
to see the error on a small console screen. 12.0 or CURRENT users will
also have to explicitly use the Forth loader instead of the Lua loader.
The error will look something like:
Warning: syntax error on file /boot/loader.conf.local
foo="xxxxxxxxxxxxxxnextboot_enable="YES"
^
/boot/support.4th has crude file I/O buffering, which uses a buffer
'read_buffer', defined to be 80 bytes by the 'read_buffer_size'
constant. The loader first tastes nextboot.conf, reading and parsing
the first line in it for nextboot_enable="YES". If this is true, then
it reopens the file and parses it like other loader .conf files.
Unfortunately, the file I/O buffering code does not fully reset the
buffer state in the reset_line_reading word. If the last file was read
to the end, that doesn't matter; the file buffer is treated as empty
anyway. But in the nextboot.conf case, the loader will not read to the
end of file if it is over 80 bytes, and the file buffer may be reused
when reading the next file. When the file is reread, the corrupt text
may cause file parsing to abort on bad syntax (if the corrupt line has
<>2 quotes in it), the wrong variable to be set, no variable to be set
at all, or (if the splice happens to land at a line ending) something
approximating normal operation.
The bug is very old, dating back to at least 2000 if not before, and is
still present in 12.0 and CURRENT r345863 (though it is now hidden by
the Lua loader by default).
Suggested one-line attached. This does change the behavior of the
reset_line_reading word, which is exported in the line-reading
dictionary (though the export is not documented in loader man pages).
But repo history shows it was probably exported for the PNP support
code, which was never included in the loader build, and was removed 5
months ago.
One thing that puzzles me: how has this bug gone unnoticed/unfixed for
nearly 2 decades? I find it hard to believe that nobody's tried to do
something interesting with nextboot, like load a kernel and filesystem,
which is what I'm doing.
Warner Losh [Wed, 28 Jul 2021 19:47:05 +0000 (13:47 -0600)]
genoffset: simplify and rewrite in sh
genoffset used the fully generic ASSYM macro to generate the offsets
needed for the thread_lite structure. However, since these are offsets
into a structure, they will always be necessarily small and positive. As
such, just create a simple character array of the right size and use a
naming convention such that we can recover the field name, structure
name and type. Use nm -t d and sort -n to sort these into order, then
loop over the resutls to generate the thread_lite structure.
projects / FreeBSD / FreeBSD.git / log