MFC r365547: Add -z "TOS" option to ping6, to test DSCP/ECN values
ping has the option to add the (deprecated) TOS byte
using the -z option. Adding the same option, with the
same (deprecated) Traffic Class Byte (nowadays actually
DSCP and ECN fields) to ping6 to validate proper QoS
processing in network switches.
During the DCTCP improvements, use of CCF_DELACK was
removed. This change is just to rename the unused flag
bit to prevent use of it, without also re-implementing
the tcp_input and tcp_output interfaces.
Rick Macklem [Thu, 24 Sep 2020 15:34:47 +0000 (15:34 +0000)]
MFC: r365895
Fix a LOR between the NFS server and server side krpc.
Recent testing of the NFS-over-TLS code found a LOR between the mutex lock
used for sessions and the sleep lock used for server side krpc socket
structures in nfsrv_checksequence(). This was fixed by r365789.
A similar bug exists in nfsrv_bindconnsess(), where SVC_RELEASE() is called
while mutexes are held.
This patch applies a fix similar to r365789, moving the SVC_RELEASE() call
down to after the mutexes are released.
This patch fixes the problem by moving the SVC_RELEASE() call in
nfsrv_bindconnsess() down a few lines to below where the mutexes are released.
John Baldwin [Wed, 23 Sep 2020 21:57:29 +0000 (21:57 +0000)]
MFC 365278: Don't assume objects in program sections have a size of a pointer.
The size of the object at 'addr' is unknown and might be smaller than
the size of a pointer (e.g. some x86 instructions are smaller than a
pointer). Instead, just check that the address is in the bounds of
the program header.
r347996:
Replace uses of `foo.(de|en)code('hex')` with `binascii.(un)?hexlify(foo)`
Python 3 no longer doesn't support encoding/decoding hexadecimal numbers using
the `str.format` method. The backwards compatible new method (using the
binascii module/methods) is a comparable means of converting to/from
hexadecimal format.
In short, the functional change is the following:
* `foo.decode('hex')` -> `binascii.unhexlify(foo)`
* `foo.encode('hex')` -> `binascii.hexlify(foo)`
While here, move the dpkt import in `cryptodev.py` down per PEP8, so it comes
after the standard library provided imports.
PR: 237403
r348024:
Followup to r347996
Replace uses of `foo.encode("hex")` with `binascii.hexlify(foo)` for forwards
compatibility between python 2.x and python 3.
PR: 237403
r348031:
Squash deprecation warning related to array.array(..).tostring()
In version 3.2+, `array.array(..).tostring()` was renamed to
`array.array(..).tobytes()`. Conditionally call `array.array(..).tobytes()` if
the python version is 3.2+.
PR: 237403
r348042:
Fix encoding issues with python 3
In python 3, the default encoding was switched from ascii character sets to
unicode character sets in order to support internationalization by default.
Some interfaces, like ioctls and packets, however, specify data in terms of
non-unicode encodings formats, either in host endian (`fcntl.ioctl`) or
network endian (`dpkt`) byte order/format.
This change alters assumptions made by previous code where it was all
data objects were assumed to be basestrings, when they should have been
treated as byte arrays. In order to achieve this the following are done:
* str objects with encodings needing to be encoded as ascii byte arrays are
done so via `.encode("ascii")`. In order for this to work on python 3 in a
type agnostic way (as it anecdotally varied depending on the caller), call
`.encode("ascii")` only on str objects with python 3 to cast them to ascii
byte arrays in a helper function name `str_to_ascii(..)`.
* `dpkt.Packet` objects needing to be passed in to `fcntl.ioctl(..)` are done
so by casting them to byte arrays via `bytes()`, which calls
`dpkt.Packet__str__` under the covers and does the necessary str to byte array
conversion needed for the `dpkt` APIs and `struct` module.
In order to accomodate this change, apply the necessary typecasting for the
byte array literal in order to search `fop.name` for nul bytes.
This resolves all remaining python 2.x and python 3.x compatibility issues on
amd64. More work needs to be done for the tests to function with i386, in
general (this is a legacy issue).
Rick Macklem [Wed, 23 Sep 2020 01:49:50 +0000 (01:49 +0000)]
MFC: r365789
Fix a LOR between the NFS server and server side krpc.
Recent testing of the NFS-over-TLS code found a LOR between the mutex lock
used for sessions and the sleep lock used for server side krpc socket
structures.
The code in nfsrv_checksequence() would call SVC_RELEASE() with the mutex
held. Normally this is ok, since all that happens is SVC_RELEASE()
decrements a reference count. However, if the socket has just been shut
down, SVC_RELEASE() drops the reference count to 0 and acquires a sleep
lock during destruction of the server side krpc structure.
This patch fixes the problem by moving the SVC_RELEASE() call in
nfsrv_checksequence() down a few lines to below where the mutex is released.
The pidx argument of isc_rxd_flush() indicates which is the last valid
receive descriptor to be used by the NIC. However, current code has
multiple issues:
- Intel drivers write pidx to their RDT register, which means that
NICs will only use the descriptors up to pidx-1 (modulo ring size N),
and won't actually use the one pointed by pidx. This does not break
reception, but it is anyway confusing and suboptimal (the NIC will
actually see only N-2 descriptors as available, rather than N-1).
Other drivers (if_vmx, if_bnxt, if_mgb) adhere to this semantic.
- The semantic used by Intel (RDT is one descriptor past the last
valid one) is used by most (if not all) NICs, and it is also used
on the TX side (also in iflib). Since iflib is not currently
using this semantic for RX, it must decrement fl->ifl_pidx
(modulo N) before calling isc_rxd_flush(), and then the
per-driver callback implementation must increment the index
again (to match the real semantic). This is confusing and suboptimal.
- The iflib refill function is also called at initialization.
However, in case the ring size is smaller than 128 (e.g. if_mgb),
the refill function will actually prepare all the receive
descriptors (N), without leaving one unused, as most of NICs assume
(e.g. to avoid RDT to overrun RDH). I can speculate that the code
looks like this right now because this issue showed up during
testing (e.g. with if_mgb), and it was easy to workaround by
decrementing pidx before isc_rxd_flush().
The goal of this change is to simplify the code (removing a bunch
of instructions from the RX fast path), and to make the semantic of
isc_rxd_flush() consistent across drivers. To achieve this, we:
- change the semantics of the pidx argument to the usual one (that
is the index one past the last valid one), so that both iflib and
drivers avoid the decrement/increment dance.
- fix the initialization code to prepare at most N-1 descriptors.
This changeset introduces the new libnetmap library for writing
netmap applications.
Before libnetmap, applications could either use the kernel API
directly (e.g. NIOCREGIF/NIOCCTRL) or the simple header-only-library
netmap_user.h (e.g. nm_open(), nm_close(), nm_mmap() etc.)
The new library offers more functionalities than netmap_user.h:
- Support for complex netmap options, such as external memory
allocators or per-buffer offsets. This opens the way to future
extensions.
- More flexibility in the netmap port bind options, such as
non-numeric names for pipes, or the ability to specify the netmap
allocator that must be used for a given port.
- Automatic tracking of the netmap memory regions in use across the
open ports.
At the moment there is no man page, but the libnetmap.h header file
has in-depth documentation.
Ian Lepore [Tue, 22 Sep 2020 14:59:05 +0000 (14:59 +0000)]
MFC r365729:
Add product ID strings for a couple Microchip usb hubs. Also, update the
vendor ID string to say just "Microchip Technology" -- the buyout of
Standard Microsystems happened in 2012 and the SMC/SMSC names are pretty
much retired at this point.
MFC r365829, r365837, r365852: certctl rehash upon install/distribute
r365829:
installworld: run `certctl rehash` after installation completes
This was originally introduced back in r360833, and subsequently reverted
because it was broken for -DNO_ROOT builds and it may not have been the
correct place for it.
While debatably this may still not be 'the correct place,' it's much cleaner
than scattering rehashes all throughout the tree. brooks has fixed the issue
with -DNO_ROOT by properly writing to the METALOG in r361397.
Do note that this is different than what was originally committed; brooks
had revisions in D24932 that made it actually use the revised unprivileged
mode and write to METALOG, along with being a little more friendly to
foreign crossbuilds and just using the certctl in-tree.
With this change, I believe we should now have a populated /etc/ssl/certs in
the VM images.
r365837:
Promote the installworld `certctl rehash` to distributeworld
Contrary to my belief, installworld is not sufficient for getting certs
installed into VM images. Promote the rehash to both installworld and
distributeworld (notably: not stageworld) and rehash the base distdir so we
end up with /etc/ssl/certs populated in the base dist archive. A future
commit will remove the rehash from bsdinstall, which doesn't really need to
happen if they're installed into base.txz.
While here, fix a minor typo: s/CERTCLTFLAGS/CERTCTLFLAGS/
r365852:
Revert r361257: bsdinstall: do a `certctl rehash` upon installation [...]
As of r365829, any given base distribution set will now include the /etc/ssl
symlinks that this rehash would've otherwise installed. This extra step is
no longer required.
MFC r365237:
Micro optimise _callout_stop_safe() by removing dead code.
The CS_DRAIN flag cannot be set at the same time like the async-drain function
pointer is set. These are orthogonal features. Assert this at the beginning
of the function.
Before:
if (flags & CS_DRAIN) {
/* FALLTHROUGH */
} else if (xxx) {
return yyy;
}
if (drain) {
zzz = drain;
}
After:
if (flags & CS_DRAIN) {
/* FALLTHROUGH */
} else if (xxx) {
return yyy;
} else {
if (drain) {
zzz = drain;
}
}
r363887:
Use the right factor when finding the best frequency and compare the
absolute value of the result.
Submitted by: kibab
r365395:
aw_clk_nm: fix incorrect use of abs()
abs() takes a (signed) int as input.
Instead, it was used with unsigned 64-bit integers.
So, add and use a new helper function to calculate a difference between
two uint64_t-s.
Rick Macklem [Mon, 21 Sep 2020 00:50:32 +0000 (00:50 +0000)]
MFC: r365703
Fix a case where the NFSv4.0 server might crash if delegations are enabled.
asomers@ reported a crash on an NFSv4.0 server with a backtrace of:
kdb_backtrace
vpanic
panic
nfsrv_docallback
nfsrv_checkgetattr
nfsrvd_getattr
nfsrvd_dorpc
nfssvc_program
svc_run_internal
svc_thread_start
fork_exit
fork_trampoline
where the panic message was "docallb", which indicates that a callback
was attempted when the ClientID is unconfirmed.
This would not normally occur, but it is possible to have an unconfirmed
ClientID structure with delegation structure(s) chained off it if the
client were to issue a SetClientID with the same "id" but different
"verifier" after acquiring delegations on the previously confirmed ClientID.
The bug appears to be that nfsrv_checkgetattr() failed to check for
this uncommon case of an unconfirmed ClientID with a delegation structure
that no longer refers to a delegation the client knows about.
This patch adds a check for this case, handling it as if no delegation
exists, which is the case when the above occurs.
Although difficult to reproduce, this change should avoid the panic().
MFC r365830: make it possible recovering broken GPT after some LBAs cut off
If pre-formatted device has GPT and a partition covering
last available LBAs and the device is attached using
a bridge reducing amount of LBAs, then it could be not enough
forcing GEOM to use primary GPT. Also, we should make it possible
to recover GPT and this requires either deleting or resizing the partition.
This change enables "gpart delete" and "gpart resize" commands
on corrupted GPT with following "gpart recover".
It still does not allow modifying corrupted GPT without
preliminary setting sysctl kern.geom.part.check_integrity=0
Alan Somers [Sat, 19 Sep 2020 19:48:15 +0000 (19:48 +0000)]
MFC r365391, r365415
r365391:
nsswitch.conf(5): recommend placing cache after files
When cache precedes files, and nscd is configured to allow negative caching,
commands like "pw groupadd" can fail. The sequence of events looks like:
1. A command like pkg(8) looks up the group, and finds it absent.
2. pkg invokes pw(8) to add the group
3. pkg queries the group, but nscd says it doesn't exist, since it has a
negative cache entry for that group.
See also: https://lists.freebsd.org/pipermail/freebsd-current/2012-January/031595.html
The first issue was lack of quoting around INSTALLFLAGS, which set it
incorrectly and produced an error on -M.
The second issue was that we weren't actually doing the install in
unprivileged mode, making it effectively useless. This was designed to pass
through the proper metalog/unpriv flags to install(1), so just let it
happen.
Allan Jude [Thu, 17 Sep 2020 21:51:05 +0000 (21:51 +0000)]
Update naming of per-dataset counters introduced in r365689
Upstream OpenZFS is changing the structure of the sysctl mibs to avoid
having a / in the element name. Chase this change here to avoid introducing
a new sysctl that will change named between 12.2 and 13.0
MFC-with: 365689
Relnotes: yes (fix example)
Sponsored by: Klara Inc.
MFC r365646, r365720:
r365646:
Enclose BRANCH_OVERRIDE in quotes in order to fix an issue with
freebsd-update(8) builds, where BRANCH is suffixed with -p0 for
builds.
r365720 (gordon):
Partially revert r346018 and use the if/then construct instead of
shell.
This would allow interested parties to do experimental runs with an
environment set appropriately to raise all the warnings throughout the
build; e.g. env WARNS=6 NO_WERROR=yes buildworld.
Not currently touching the numerous instances in ^/tools.
MFC r365493-r365494, r365600, r365602, r365637: various WARNS fixes
r365493:
libc/resolv: attempt to fix the test under WARNS=6
In a side-change that I'm working on to start defaulting src builds to
WARNS=6 where WARNS isn't otherwise specified, GCC6 (and clang, to a lesser
extent) pointed out a number of issues with the resolv tests:
- Global method variable that gets shadowed in run_tests()
- Signed/unsigned comparison between i in run_tests() and hosts->sl_cur
The shadowed variable looks like it might actually be bogus as written, as
we pass it to RUN_TESTS -> run_tests, but other parts use the global method
instead. This change is mainly geared towards correcting that by removing
the global and plumbing the method through from run_tests -> run into the
new thread.
For the signed/unsigned comparison, there's no compelling reason to not just
switch i/nthreads/nhosts to size_t.
The review also included a change to the load() function that was better
addressed by jhb in r365302.
r365494:
libc tests: dynthr_mod: fix some WARNS issues
This is being addressed as part of a side-patch I'm working on that builds
all the things with WARNS=6, instead of relying on it being supplied in just
shallow parts of the build with higher-level Makefile.inc.
Provide a prototype for mod_main and annotate the thread function argument
as unused.
r365600:
MFV r365599: import fix for a libexecinfo warning at higher WARNS
v1.17 of this file included a fix that I just submitted upstream to fix a
warning about prevent_inline with external linkage not having been
previously declared.
r365602:
librt: tests: fix minor issues with higher WARNS
got_sigalrm is a global with external linkage and must therefore have a
previous extern declaration. There's no reason to maintain the status quo
there, so just make it static.
The result var is unused.
This part of the test has not been upstreamed, presumably because it exists
solely for sem_clockwait_np. We should perhaps consider moving it into its
own test file outside of ^/contrib/netbsd-tests, but this can happen later.
r365637:
MFV r365636: libarchive: import fix for WARNS=6 builds in testing bits
Two more cases of explicitly marking globals for internal linkage where they
need not be shared. Committed upstream as of a38e62314a1f.
John Baldwin [Wed, 16 Sep 2020 22:55:27 +0000 (22:55 +0000)]
MFC 365276: Compute the correct size of the string to move forward.
Previously this was counting the amount of spare room at the start of
the buffer that the string needed to move forward and passing that as
the number of bytes to copy to memmove rather than the length of the
string to be copied.
In the strfmon test in the test suite this caused the memmove to
overflow the allocated buffer by one byte which CHERI caught.
Remove documentation of RELEASE_CRUNCH here. It's obsolete and no longer a
good example.
r364166:
Fix crunchgen usage of mkstemp()
On Glibc systems mkstemp can only be used once with the same template
string since it will be modified in-place and no longer contain any 'X'
chars. It is fine to reuse the same file here but we need to be explicit and
use open() instead of mkstemp() on the second use.
While touching this file also avoid a hardcoded /bin/pwd since that may not
work when building on non-FreeBSD systems.
r364174:
Use env pwd instead of pwd in crunchgen.c
In r364166 I changed /bin/pwd to pwd, but pwd can be shell builtin that
may not correctly return a real path. To ensure that all symlinks are
resolved use `env pwd -P` instead (the -P flag is part of POSIX so
should be supported everywhere).
r364234:
crunchgen: use pwd -P without env
The -P flag is required by POSIX so we don't have to care whether pwd is
a shell builtin or not. This also allows removing pwd from the list of
bootstrap tools since all shells we care about for building have a
builtin pwd command. This effectively reverts r364190.
r364646:
Re-indent crunched_main.c in preparation for D25998
r364647:
Correctly determine the real executable in crunched binaries
This should fix cases like su setting argv[0] to _su for /bin/sh.
Previously cheribsdbox (a crunched tool we use in CheriBSD to reduce the
size of our minimal disk images to allow loading them onto FPGAs without
waiting forever for the transfer) would complain about _su not being
compiled in, but now that we also look at AT_EXECPATH it correctly
invokes the sh tool.
Note: we use use AT_EXECPATH instead of the KERN_PROC_PATHNAME sysctl to get
the crunchgen binary name since it seems like KERN_PROC_PATHNAME just
returns the last cached path for a given hardlink.
When using `su`, instead of invoking /bin/csh this would invoke the last
used hardlink to cheribsdbox. This caused weird test failures when running
tests due to `id` being executed instead of `echo`:
$ id # id is a hardlink to /bin/cheribsdbox
$ su postgres -c 'echo 1' # su is also a hardlink
uid=1001(postgres) gid=1001(postgres) groups=1001(postgres)
r365605:
crunchgen(8): fix crunched application build with WARNS=6
This was revealed by the rescue build with a patch I'm working on to default
WARNS=6 everywhere. The issues resolved were:
- Missing prototype for _crunched_${ident}_stub in the *_stub.c generated
bits
- Missing prototype for crunched_main
- Incomplete prototype for _crunched_${ident}_stub in the generated parts of
crunched_main
- Literal strings in the stub table must drop const qualifier, unless we
const'ify name
- f field in struct stub didn't have a proper prototype
Most of these issues are minor formalities and easily addressed.
I note that if my patch to eventually raise WARNS for the rescue build
lands, we'll need to bump the __FreeBSD_version requirement for
bootstrapping crunchgen and wipe out the rescue .OBJDIR if it's stale, which
we should be able to detect pretty easily from a couple of the issues that
have been fixed here.
r365705:
__FreeBSD_version bump for r365605 (crunchgen producing WARNS-clean)
The change in D26397 will need a __FreeBSD_version to base off of for
bootstrapping crunchgen, to avoid avoidable build failures just because the
host has an outdated crunchgen.
o Enhance dependency loop logging: print full chain instead of the
last link competing the loop;
o Add -g option to generate dependency graph suitable for GraphViz
visualization, loops and other graph generation issues are highlighted
automatically;
o Add -p option that enables grouping items that can be processed in
parallel.
Submitted by: Boris Lytochkin <lytboris at gmail>
Differential Revision: https://reviews.freebsd.org/D25389
Allan Jude [Sun, 13 Sep 2020 16:28:25 +0000 (16:28 +0000)]
MFOpenZFS: Introduce read/write kstats per dataset
The following patch introduces a few statistics on reads and writes
grouped by dataset. These statistics are implemented as kstats
(backed by aggregate sums for performance) and can be retrieved by
using the dataset objset ID number. The motivation for this change is
to provide some preliminary analytics on dataset usage/performance.
Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Matthew Ahrens <mahrens@delphix.com> Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
openzfs/zfs@a448a2557ec4938ed6944c7766fe0b8e6e5f6456
Also contains parts of:
MFOpenZFS: Connect dataset_kstats for FreeBSD
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Reviewed by: Sean Eric Fagan <sef@ixsystems.com> Reviewed-by: Serapheim Dimitropoulos <serapheim@delphix.com> Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Allan Jude <allan@klarasystems.com>
openzfs/zfs@4547fc4e071ceb1818b3a46c3035b923e06e5390
This is a direct commit to stable/12 because they do not exist in
illumos upstream ZFS and needed to be heavily modified to work in
stable/12
getlogin_r is specified by POSIX to to take a size_t len, not int. Fix our
version to do the same, bump the symbol version due to ABI change and
provide compat.
This was reported to break compilation of Ruby 2.8.
Some discussion about the necessity of the ABI compat did take place in the
review. While many 64-bit platforms would likely be passing it in a 64-bit
register and zero-extended and thus, not notice ABI breakage, some do
sign-extend (e.g. mips).
MFC r365500: certctl: fix hashed link generation with duplicate subjects
Currently, certctl rehash will just keep clobbering .0 rather than
incrementing the suffix upon encountering a duplicate. Do this, and do it
for blacklisted certs as well.
This also improves the situation with the blacklist to be a little less
flakey, comparing cert fingerprints for all certs with a matching subject
hash in the blacklist to determine if the cert we're looking at can be
installed.
Future work needs to completely revamp the blacklist to align more with how
it's described in PR 246614. In particular, /etc/ssl/blacklisted should go
away to avoid potential confusion -- OpenSSL will not read it, it's
basically certctl internal.
Follow-up r365662 (MFC of r365371 and r365373) by also removing the
header hack from jemalloc_FreeBSD.h, which rendered any make.conf
MALLOC_PRODUCTION or src.conf WITH/WITHOUT_MALLOC_PRODUCTION irrelevant.
Direct commit to stable/{11,12} as this does not apply to head.
Follow-up r365662 (MFC of r365371 and r365373) by correctly setting
WITH_MALLOC_PRODUCTION for stable branches. Also add a note to UPDATING,
to inform users about the new setting.
Direct commit to stable/{11,12} as this does not apply to head.
Noticed by: imp, Ronald Klop <ronald-lists@klop.ws>
Turn MALLOC_PRODUCTION into a regular src.conf(5) option
For historical reasons, defining MALLOC_PRODUCTION in /etc/make.conf has
been used to turn off potentially expensive debug checks and statistics
gathering in the implementation of malloc(3).
It seems more consistent to turn this into a regular src.conf(5) option,
e.g. WITH_MALLOC_PRODUCTION / WITHOUT_MALLOC_PRODUCTION. This can then
be toggled similar to any other source build option, and turned on or
off by default for e.g. stable branches.
There have been several mentions on our mailing lists about missing
atomic functions in our system libraries (e.g. __atomic_load_8 and
friends), and recently I saw __bswapdi2 and __bswapsi2 mentioned too.
To address this, add implementations for the functions from compiler-rt
to the system compiler support libraries, e.g. libcompiler_rt.a and and
libgcc_s.so.
This also needs a small fixup in compiler-rt's atomic.c, to ensure that
32-bit mips can build correctly.
Bump __FreeBSD_version to make it easier for port maintainers to detect
when these functions were added.
After r364753, there should be no need to suppress -Watomic-alignment
warnings anymore for compiler-rt's atomic.c. This occurred because the
IS_LOCK_FREE_8 macro was not correctly defined to 0 for mips, and this
caused the compiler to emit a runtime call to __atomic_is_lock_free(),
and that triggers the warning.
MFC r365509:
Follow-up r364753 by enabling compiler-rt's atomic implementation only
for clang, as it uses clang specific builtins, and does not compile
correctly with gcc. Note that gcc packages usually come with their own
libatomic, providing these primitives.
MFC r365588:
Follow-up r364753 by only using arm's stdatomic.c implementation, as it
already covers the functions in compiler-rt's atomic.c, leading to
conflicts when linking.
There's a race where dying vnets move their interfaces back to their original
vnet, and if_epair cleanup (where deleting one interface also deletes the other
end of the epair). This is commonly triggered by the pf tests, but also by
cleanup of vnet jails.
As we've not yet been able to fix the root cause of the issue work around the
panic by not dereferencing a NULL softc in epair_qflush() and by not
re-attaching DYING interfaces.
This isn't a full fix, but makes a very common panic far less likely.
projects / FreeBSD / FreeBSD.git / log