Warner Losh [Thu, 15 Jul 2021 22:46:06 +0000 (16:46 -0600)]
awk: revert upstream's attempt to disallow hex strings
Upstream one-true-awk decided to disallow hex strings as numbers. This
is in line with awk's behavior prior to C99, and allowed by the POSIX
standard. The standard, however, allows them to be treated as numbers
because that's what the standard said in the 2001 through 2004 editions.
Since 2001, the nawk in FreeBSD has treated them as numbers, so restore
that behavior, allowed by the standard.
A number of scripts in the FreeBSD tree depend on this interpretation,
including scripts to build the kernel which had mysteriously started
failing for some people and not others. By re-allowing 0x hex numbers,
this fixes those scripts and restores POLA.
Upstream issue: https://github.com/onetrueawk/awk/issues/126
Sponsored by: Netflix
Reviewed by: kevans
MFC After: asap due to regression alrady merged to stable
Differential Revision: https://reviews.freebsd.org/D31199
pf: allow table stats clearing and reading with ruleset rlock
Instead serialize against these operations with a dedicated lock.
Prior to the change, When pushing 17 mln pps of traffic, calling
DIOCRGETTSTATS in a loop would restrict throughput to about 7 mln. With
the change there is no slowdown.
Creating tables and zeroing their counters induces excessive IPIs (14
per table), which in turns kills single- and multi-threaded performance.
Work around the problem by extending per-CPU counters with a general
counter populated on "zeroing" requests -- it stores the currently found
sum. Then requests to report the current value are the sum of per-CPU
counters subtracted by the saved value.
Sample timings when loading a config with 100k tables on a 104-way box:
stock:
pfctl -f tables100000.conf 0.39s user 69.37s system 99% cpu 1:09.76 total
pfctl -f tables100000.conf 0.40s user 68.14s system 99% cpu 1:08.54 total
patched:
pfctl -f tables100000.conf 0.35s user 6.41s system 99% cpu 6.771 total
pfctl -f tables100000.conf 0.48s user 6.47s system 99% cpu 6.949 total
Stefan Eßer [Sat, 10 Jul 2021 11:00:56 +0000 (13:00 +0200)]
libalias: fix divide by zero causing panic
The packet_limit can fall to 0, leading to a divide by zero abort in
the "packets % packet_limit".
An possible solution would be to apply a lower limit of 1 after the
calculation of packet_limit, but since any number modulo 1 gives 0,
the more efficient solution is to skip the modulo operation for
packet_limit <= 1.
Michael Tuexen [Fri, 11 Jun 2021 18:14:34 +0000 (20:14 +0200)]
tcp: tolerate missing timestamps
Some TCP stacks negotiate TS support, but do not send TS at all
or not for keep-alive segments. Since this includes modern widely
deployed stacks, tolerate the violation of RFC 7323 per default.
Alexander Motin [Tue, 6 Jul 2021 02:19:48 +0000 (22:19 -0400)]
nvme(4): Report NPWA before NPWG as stripesize.
New Samsung 980 SSDs report Namespace Preferred Write Alignment of
8 (4KB) and Namespace Preferred Write Granularity of 32 (16KB).
My quick tests show that 16KB is a minimal sequential write size
when the SSD reaches peak IOPS, so writing much less is very slow.
But writing slightly less or slightly more does not change much,
so it seems not so much a size granularity as minimum I/O size.
Thinking about different stripesize consumers:
- Partition alignment should be based on NPWA by definition.
- ZFS ashift in part of forcing alignment of all I/Os should also
be based on NPWA. In part of forcing size granularity, if really
needed, it may be set to NPWG, but too big value can make ZFS too
space-inefficient, and the 16KB is actually the biggest supported
value there now.
- ZFS recordsize/volblocksize could potentially be tuned up toward
NPWG to work as I/O size granularity, but enabled compression makes
it too fuzzy. And those are normally user-configurable things.
- ZFS I/O aggregation code could definitely use Optimal Write Size
value and may be NPWG, but we don't have fields in GEOM now to report
the minimal and optimal I/O sizes, and even maximal is not reported
outside GEOM DISK to be used by ZFS.
Import the latest bsd-features branch of the one-true-awk upstream:
o Move to bison for $YACC
o Set close-on-exec flag for file and pipe redirects that aren't std*
o lots of little fixes to modernize ocde base
o free sval member before setting it
o fix a bug where a{0,3} could match aaaa
o pull in systime and strftime from NetBSD awk
o pull in fixes from {Net,Free,Open}BSD (normalized our code with them)
o add BSD extensions and, or, xor, compl, lsheift, rshift (mostly a nop)
Also revert a few of the trivial FreeBSD changes that were done slightly
differently in the upstreaming process. Also, our PR database may have
been mined by upstream for these fixes, and Mikolaj Golub may deserve
credit for some of the fixes in this update.
Alex Richardson [Mon, 21 Sep 2020 19:03:07 +0000 (19:03 +0000)]
awk: Fix subobject out-of-bounds access
When matching a regex with ^, it would attempt to access
gototab[NSTATES][NCHARS+2], and therefore access the state for the \002
character instead. This change is required to run awk under CHERI (with
sub-object bounds) and when running with UBSan instrumentation.
Warner Losh [Mon, 3 Jun 2019 05:25:22 +0000 (05:25 +0000)]
Another partial revert of r301289.
In this case, a change was made in one-true-awk from *FS to
getsval(fsloc) in a line just after one of the lines that had the 0 ->
NULL change. It works both ways as far as I can tell. It looks like a
bug fix, but I've not tried to track down which ancient version of
one-true-awk it was in (github starts too late for tracking this
down). Before and after the changes the regression suite is passes
100% relative to the un-modified one-true-awk.
Warner Losh [Mon, 3 Jun 2019 05:25:16 +0000 (05:25 +0000)]
Fix mismerge that crept into r301289.
The conversion of 0 -> NULL required a rebase at some point, as noted
in r301289 when pfg commited it. In that rebase, three lines remained
that had been removed in a prior version of awk, and one of them had a
0 -> NULL change causing a conflict. The conflict should have been
resolved by removing the three lines, but wasn't. This introduces a
regression into f.split3 test which prior to this commit we were
failing, but a pure onetrueawk wasn't. Remove the offending 3 lines.
Warner Losh [Sun, 2 Jun 2019 16:25:07 +0000 (16:25 +0000)]
Merge from upstream at 4189ef5d from https://github.com/onetrueawk/awk.git
Note: this backs out a number of changes we've made to awk because
they aren't upstream, but are on the vendor branch. Those will be
reapplied. svn makes it needlessly difficult to know which ones, but
at least r315426, r301289, and maybe r301691, though there may be
others too. None of these are critical, so bisecting through this
point is safe for all but awk regression tests :).
Chuck Tuffli [Sun, 27 Jun 2021 22:14:52 +0000 (15:14 -0700)]
bhyve: Fix NVMe iovec construction for large IOs
The UEFI driver included with Rocky Linux 8.4 uncovered an existing bug
in the NVMe emulation's construction of iovec's.
By default, NVMe data transfer operations use a scatter-gather list in
which all entries point to a fixed size memory region. For example, if
the Memory Page Size is 4KiB, a 2MiB IO requires 512 entries. Lists
themselves are also fixed size (default is 512 entries).
Because the list size is fixed, the last entry is special. If the IO
requires more than 512 entries, the last entry in the list contains the
address of the next list of entries. But if the IO requires exactly 512
entries, the last entry points to data.
The NVMe emulation missed this logic and unconditionally treated the
last entry as a pointer to the next list. Fix is to check if the
remaining data is greater than the page size before using the last entry
as a pointer to the next list.
Kristof Provost [Tue, 29 Jun 2021 08:26:40 +0000 (10:26 +0200)]
pf: Reduce the data returned in DIOCGETSTATESNV
This call is particularly slow due to the large amount of data it
returns. Remove all fields pfctl does not use. There is no functional
impact to pfctl, but it somewhat speeds up the call.
It might affect other (i.e. non-FreeBSD) code that uses the new
interface, but this call is very new, so there's unlikely to be any. No
releases contained the previous version, so we choose to live with the
ABI modification.
Kristof Provost [Mon, 28 Jun 2021 10:48:20 +0000 (12:48 +0200)]
pf tests: Stress state retrieval
Create and retrieve 20.000 states. There have been issues with nvlists
causing very slow state retrieval. We don't impose a specific limit on
the time required to retrieve the states, but do log it. In excessive
cases the Kyua timeout will fail this test.
Warner Losh [Fri, 21 Jun 2019 03:49:36 +0000 (03:49 +0000)]
Mount and unmount devfs around calls to add packages.
pkg now uses /dev/null for some of its operations. NanoBSD's packaging
stuff didn't mount that for the chroot it ran in, so any config that
added packages would see the error:
pkg: Cannot open /dev/null:No such file or directory
when trying to actually add those packages. It's easy enough for
nanobsd to mount /dev and it won't hurt anything that was already
working and may help things that weren't (like this). I moved the
mount/unmount pair to be in the right push/pop order from the
submitted patch.
PR: 238727
Submitted by: mike tancsa
Tested by: Karl Denninger
Warner Losh [Wed, 18 Nov 2020 19:22:24 +0000 (19:22 +0000)]
mergemaster: handle symbolic links during update.
/etc/os-release is now a symbolic link to a generated file. Make
mergemaster cope with symbolic links generically. I'm no longer
a big mergemaster user, so this has only been lightly tested
by me, though Kimura-san has ran it through its paces.
Olivier Houchard [Wed, 30 Jun 2021 20:56:50 +0000 (22:56 +0200)]
arm: Make sure we can handle a thumb entry point.
Similarly to what's been done on arm64 with commit 712c060c94fd447c91b0e6218c12a431206b487a, when executing a binary, if the
entry point is a thumb symbol, then make sure we set the PSL_T flag, otherwise
the CPU will interpret it in ARM mode, and that will likely leads to an
undefined instruction.
Ed Maste [Sun, 27 Jun 2021 17:21:26 +0000 (13:21 -0400)]
Add deprecation notice for WITH_PROFILE option
As discussed on freebsd-current [1] and freebsd-arch [2] and review
D30833, FreeBSD 14 will ship without the _p.a libraries built with -pg.
Both upstream and base system (in commit b762974cf4b9) Clang have been
modified to remove the special case for linking against these libraries.
Clang's -pg support and mcount() remain, so building with -pg can still
be used on code that the user builds; we just do not provide prebuilt
libraries compiled with -pg. A similar change is still needed for GCC.
Mariusz Zaborski [Fri, 11 Jun 2021 15:35:36 +0000 (17:35 +0200)]
libnv: optimize nvlist size calculation
If we had a multiple nvlist, during nvlist_pack, we calculated the size
of every nvlist separately. For example, if we had a nvlist with three
nodes each containing another (A contains B, and B contains C), we first
calculated the size of nvlist A (which contains B, C), then we calculate
the size of B (which contains C, notice that we already did the
calculation of B, when we calculate A), and finally C. This means that
this calculation was O(N!). This was done because each time we pack
nvlist, we have to put its size in the header
(the separate header for A, B, and C).
To not break the ABI and to reduce the complexity of nvlist_size,
instead of calculating the nvlist size when requested,
we track the size of each nvlist.
Mariusz Zaborski [Sun, 17 Feb 2019 18:26:27 +0000 (18:26 +0000)]
libnv: fix double free
In r343986 we introduced a double free. The structure was already
freed fixed in the r302966. This problem was introduced
because the GitHub version was out of sync with the FreeBSD one.
Submitted by: Mindaugas Rasiukevicius <rmind@netbsd.org>
MFC with: r343986
Mariusz Zaborski [Sun, 10 Feb 2019 23:28:55 +0000 (23:28 +0000)]
libnv: fix memory leaks
nvpair_create_stringv: free the temporary string; this fix affects
nvlist_add_stringf() and nvlist_add_stringv().
nvpair_remove_nvlist_array (NV_TYPE_NVLIST_ARRAY case): free the chain
of nvpairs (as resetting it prevents nvlist_destroy() from freeing it).
Note: freeing the chain in nvlist_destroy() is not sufficient, because
it would still leak through nvlist_take_nvlist_array(). This affects
all nvlist_*_nvlist_array() use
Submitted by: Mindaugas Rasiukevicius <rmind@netbsd.org>
Reported by: clang/gcc ASAN
MFC after: 2 weeks
sbin/md5: improve compatibility with coreutils -c option
The -c option expects a digest file in either BSD or coreutils format.
The output for matched and mismatched files is identical to that
of the coreutils version.
The review of these changes included test cases that have already
been committed for the functionality that existed before.
Another test script is added to cover the coreutils compatible
extension implemented by this patch.
This commit contains a tests/Makefile that has been cleaned up
compared to the review version, using an implicit rule to apply the
TESTBASE path at build time (and the scripts have been renamed to
have an extension of .SH instead of .sh to trigger this rule).
md5: Improve compatibility with coreutils and format fix
The previous changes that added support for the coreutils -c option
modified the output generated by passing -r to match that of the
coreutils versions. The difference is that coreutils separates the
hash from the file name by two blanks " " (or a blank followed by
an asterisk " *" with the -b option denoting).
While most scripts or users will not notice the difference, it might
be considered a violation of POLA and this commit reverts the change
for the non-sum programs. These will print a single blank " " as the
separator, as they die before the previous commit.
In order to still generate output that is identical to that of the
coreutils programs, this commit generates the " " or " *" separator
used by them for the -sum versions, depending on the presence of the
-b option.
Warner Losh [Thu, 20 May 2021 17:26:46 +0000 (11:26 -0600)]
md5: portability fix -- include stdbool.h explicitly
stdbool.h needs to be included to use type bool variables. Due to
namespace pollution, this gets brought in on FreeBSD, but not on
other systems. Include it explicilty.
Warner Losh [Wed, 19 May 2021 17:26:20 +0000 (11:26 -0600)]
md5: Create md5sum, etc compatible programs
On Linux, there's a similar set of programs to ours, but that end in the
letters 'sum'. These act basically like FreeBSD versions run with the -r
option. Add code so that when the program ends in 'sum' you get the
linux -r behavior. This is enough to make most things that use sha*sum
work correctly (the -c / --check options, as well as the long args are
not implemented). When running with the -sum programs, ignore -t instead
of running internal speed tests and make -c an error.
Reviewed by: sef, and kp and allanjude (earlier version)
Relnotes: yes
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D30309
While the correctness of the supported hash algorithms can be tested
with the built-in self-test feature, these test cases are meant to
detect changes in the output format.
A follow-up commit will improve the compatibility with the coreutils
versions of the hash programs, and these tests should detect any
unintended side-effects of such a change on existing functionality.
sbin/md5: fix test case to not write outside temp directory
The bsd-p-test did create files 1.out .. 8.out in the test directory.
This has been fixed and the temporary output files are now written
to the temporary directory.
Lutz Donnerhacke [Thu, 27 May 2021 21:42:54 +0000 (23:42 +0200)]
libalias: Switch to SPLAY trees
Current data structure is using a hash of unordered lists. Those
unordered lists are quite efficient, because the least recently
inserted entries are most likely to be used again. In order to avoid
long search times in other cases, the lists are hashed into many
buckets. Unfortunatly a search for a miss needs an exhaustive
inspection and a careful definition of the hash.
Splay trees offer a similar feature - almost O(1) for access of the
least recently used entries), and amortized O(ln(n) - for almost all
other cases. Get rid of the hash.
Now the data structure should able to quickly react to external
packets without eating CPU cycles for breakfast, preventing a DoS.
Lutz Donnerhacke [Tue, 25 May 2021 13:03:34 +0000 (15:03 +0200)]
libalias: Restructure
Clean up the database handling in order to switch to more efficient
data structures. The development of this patch was artificially split
in to many small steps to ease reviewing.
- Common search terms
- Separate fully qualified search
- Separate table for partial links
- Cleanup _FindLinkIn
- Factor out the outgoing search function
- Factor out a common idiom to return found links
- Reorder incoming links by grouping of common search terms
- Remove LSNAT from outgoing search
- Group internal structure semantically
- Separate table for PPTP
- Use AliasRange instead of PORT_BASE
- Remove temporary state deleteAllLinks from global struct
- Avoid uninitialized expiration
Lutz Donnerhacke [Tue, 25 May 2021 07:07:21 +0000 (09:07 +0200)]
libalias: Promote per instance global variable timeStamp
Summary:
- Use LibAliasTime as a real global variable for central timekeeping.
- Reduce number of syscalls in user space considerably.
- Dynamically adjust the packet counters to match the second resolution.
- Only check the first few packets after a time increase for expiry.
Lutz Donnerhacke [Sat, 15 May 2021 15:35:36 +0000 (17:35 +0200)]
libalias: tidy up housekeeping
Replace current expensive, but sparsly called housekeeping
by a single, repetive action.
This is part of a larger restructure of libalias in order to switch to
more efficient data structures. The whole restructure process is
split into 15 reviews to ease reviewing. All those steps will be
squashed into a single commit for MFC in order to hide the
intermediate states from production systems.
As for example pfctl -ss keeps calling it, it saves a lot of overhead
from elided parsing of /etc/nsswitch.conf and /etc/protocols.
Sample result when running a pre-nvlist binary with nfs root and dumping
7 mln states:
before: 24.817u 62.993s 1:28.52 99.1%
after: 8.064u 1.117s 0:18.87 48.5%
Rick Macklem [Sat, 26 Jun 2021 23:47:59 +0000 (16:47 -0700)]
param.h: Bump __FreeBSD_version for commit 6ae32cc8182f
Commit 6ae32cc8182f changed the internal KAPI between the krpc and
nfsd modules. As such, both need to be rebuilt from sources.
Bump __FreeBSD_version to 1202508.
Rick Macklem [Fri, 11 Jun 2021 23:57:14 +0000 (16:57 -0700)]
krpc: Acquire ref count of CLIENT for backchannel use
Michael Dexter <editor@callfortesting.org> reported
a crash in FreeNAS, where the first argument to
clnt_bck_svccall() was no longer valid.
This argument is a pointer to the callback CLIENT
structure, which is free'd when the associated
NFSv4 ClientID is free'd.
This appears to have occurred because a callback
reply was still in the socket receive queue when
the CLIENT structure was free'd.
This patch acquires a reference count on the CLIENT
that is not CLNT_RELEASE()'d until the socket structure
is destroyed. This should guarantee that the CLIENT
structure is still valid when clnt_bck_svccall() is called.
It also adds a check for closed or closing to
clnt_bck_svccall() so that it will not process the callback
RPC reply message after the ClientID is free'd.
Rick Macklem [Mon, 7 Jun 2021 20:48:25 +0000 (13:48 -0700)]
nfscl: Fix generation of va_fsid for a tree of NFSv4 server file systems
Pre-r318997 the code looked like:
if (vp->v_mount->mnt_stat.f_fsid.val[0] != (uint32_t)np->n_vattr.na_filesid[0])
vap->va_fsid = (uint32_t)np->n_vattr.na_filesid[0];
Doing this assignment got lost by r318997 and, as such, NFSv4 mounts
of servers with trees of file systems on the server is broken, due to duplicate
fileno values for the same st_dev/va_fsid.
Although I could have re-introduced the assignment, since the value of
na_filesid[0] is not guaranteed to be unique across the server file systems,
I felt it was better to always do the hash for na_filesid[0,1].
Since dev_t (st_dev/va_fsid) is now 64bits, I switched to a 64bit hash.
There is a slight chance of a hash conflict where 2 different na_filesid
values map to same va_fsid, which will be documented in the BUGS
section of the man page for mount_nfs(8). Using a table to keep track
of mappings to catch conflicts would not easily scale to 10,000+ server file
systems and, when the conflict occurs, it only results in fts(3) reporting
a "directory cycle" under certain circumstances.
Alex Richardson [Tue, 30 Mar 2021 13:53:41 +0000 (14:53 +0100)]
sys/dev/md: Drop unncessary __GLOBL(mfs_root)
LLVM12 complains if you change the symbol binding:
error: mfs_root_end changed binding to STB_WEAK [-Werror,-Winline-asm]
error: mfs_root changed binding to STB_WEAK [-Werror,-Winline-asm]
Kristof Provost [Fri, 4 Jun 2021 08:52:07 +0000 (10:52 +0200)]
pf: store L4 headers in pf_pdesc
Rather than pointers to the headers store full copies. This brings us
slightly closer to what OpenBSD does, and also makes more sense than
storing pointers to stack variable copies of the headers.
Dimitry Andric [Fri, 25 Jun 2021 18:42:38 +0000 (20:42 +0200)]
Work around bogus old gcc "initializer element is not constant" error
After df3b437c1e073eb83e9a93af1c417f3ee8d0de3b, older gcc's such as
4.2.1 (still used on earlier branches for e.g. mips and powerpc) and
6.3.0 (still used for some cross-builds) started throwing bogus errors
like:
In file included from /workspace/src/lib/msun/src/s_llround.c:11:0:
/workspace/src/lib/msun/src/s_lround.c:54:31: error: initializer element is not constant
static const type dtype_min = type_min - 0.5;
^~~~~~~~
/workspace/src/lib/msun/src/s_lround.c:55:31: error: initializer element is not constant
static const type dtype_max = type_max + 0.5;
^~~~~~~~
Since 'type_min' and 'type_max' are constants declared just above these
lines this error is nonsensical, but older gcc's are not smart enough.
Work around the error by reusing the (type)DTYPE_MIN and (type)DTYPE_MAX
macros, so I can MFC this right away, unbreaking a few stable builds.
Dimitry Andric [Tue, 22 Jun 2021 16:38:27 +0000 (18:38 +0200)]
Fix failures in libm's lround_test after clang 12 import
It turned out that the (type)DTYPE_MAX conversions at the top of
s_lround.c are now emitted as cvtsi2sd instructions, at least on SSE
capable CPUs. This caused the FE_INEXACT flag to always be set, at least
for the double and float variants. Under clang 11, the whole INRANGE()
comparisons were still optimized away, but this has "improved" in clang
12, due to stricter adherence to the -ffp-exception-behavior=maytrap
compiler flag.
To avoid run-time integer to float conversions, use static constants
instead, so they are computed at compile time, and the INRANGE()
statements are optimized away again, if applicable.
While here, use an integer instead of a floating type to store the test
results in lround_test.c, as this is more appropriate, and we can also
drop the volatile hack.
Dimitry Andric [Tue, 22 Jun 2021 20:26:13 +0000 (22:26 +0200)]
Disable llvm generating 128-bit multiply libcalls on 32-bit ARM
Merge commit 789708617d20 from llvm git (Koutheir Attouchi):
Do not generate calls to the 128-bit function __multi3() on 32-bit ARM
Re-applying this patch after bots failures. Should be fine now.
The function __multi3() is undefined on 32-bit ARM, so a call to it should
never be emitted. Instead, plain instructions need to be generated to
perform 128-bit multiplications.
Mark Johnston [Wed, 19 Dec 2018 04:54:32 +0000 (04:54 +0000)]
Remove a use of a negative array index from fxp(4).
This fixes a warning seen when compiling amd64 GENERIC with clang 7.
Also remove the workaround added in r337324. clang 7 and gcc 4.2
generate the same code with or without the code change.
Dimitry Andric [Mon, 14 Jun 2021 19:17:05 +0000 (21:17 +0200)]
Export various 128 bit long double functions from libgcc_s.so.1
These were already compiled for some time on aarch64 and riscv, by
including lib/libcompiler_rt/Makefile.inc, but never exported in the
shared library. Since gcc exports these under version GCC_4.6.0, we do
the same.
This review should replace D11482 for now. For e.g. amd64 more work is
still to be done, as compiler-rt does not seem to support 128 bit long
double math for that architecture.
Reviewed by: cem
Differential Revision: https://reviews.freebsd.org/D28690
Kristof Provost [Fri, 21 May 2021 12:26:49 +0000 (14:26 +0200)]
dummynet: Fix schedlist and aqmlist locking
These are global (i.e. shared across vnets) structures, so we need
global lock to protect them. However, we look up entries in these lists
(find_aqm_type(), find_sched_type()) and return them. We must ensure
that the returned structures cannot go away while we are using them.
Resolve this by using NET_EPOCH(). The structures can be safely accessed
under it, and we postpone their cleanup until we're sure they're no
longer used.
projects / FreeBSD / FreeBSD.git / log