Prepare the cubic congestion control module to better align with
the specifications in RFC8312bis.
Rename a few cubic state variables to the variable names found in
the RFC8312bis specification. This makes the code more understandable
for someone reading the RFC and the code. It also makes the variable
naming convention more uniform. Add some variables needed subsequently.
No functional change.
Submitted By: Bhaskar Pardeshi, VMware Inc.
Reviewed By: tuexen, #transport
Sponsored by: NetApp, Inc.
Differential Revision: https://reviews.freebsd.org/D40436
tcp: make the maximum number of retransmissions tunable per VNET
Both Windows (TcpMaxDataRetransmissions) and Linux (tcp_retries2)
allow to restrict the maximum number of consecutive timer based
retransmissions. Add that same capability on a per-VNet basis to
FreeBSD.
Andrew Turner [Mon, 22 May 2023 18:52:54 +0000 (19:52 +0100)]
Switch to contigmalloc in the Hyper-V code
In the Hyper-V drivers we need to allocate buffers shared between the
host and guest. This memory has been allocated with bus_dma, however
it doesn't use this correctly, e.g. it is missing calls to
bus_dmamap_sync. Along with this on arm64 we need this memory to be
mapped with the correct memory type that bus_dma may not use.
Switch to contigmalloc to allocate this memory as this will correctly
allocate cacheable memory.
Kristof Provost [Fri, 2 Jun 2023 14:38:30 +0000 (16:38 +0200)]
netinet: re-read IP length after PFIL hook
The pfil hook may modify the packet, so before we check its length (to
decide if it needs to be fragmented or not) we should re-read that
length.
This is most likely to happen when pf is reassembling packets. In that
scenario we'd receive the last fragment, which is likely to be a short
packet, pf would reassemble it (likely exceeding the interface MTU) and
then we'd transmit it without fragmenting, because we're comparing the
MTU to the length of the last fragment, not the fully reassembled
packet.
rtld: do not allow both dynamic DTV index and static TLS offset
If we are allocating static offset for an object with dynamic index,
return failure. In the opposite case, if dynamic index is requested for
statically allocated TLS area, directly use the offset instead of
setting the index.
Taken from NetBSD Joerg Sonnenberger change for src/libexec/ld.elf_so/tls.c
rev. 1.18.
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
OPENSSL_API_COMPAT can be used to specify the OpenSSL API version in
use for the purpose of hiding deprecated interfaces and enabling
the appropriate deprecation notices.
This change is a NFC while we're still using OpenSSL 1.1.1 but will
avoid deprecation warnings upon the switch to OpenSSL 3.0.
A future update may migrate to use the OpenSSL 3.0 APIs.
PR: 271615
Reviewed by: emaste
Sponsored by: The FreeBSD Foundation
Colin Percival [Sun, 4 Jun 2023 23:01:31 +0000 (16:01 -0700)]
Revert "Revert "tslog: Annotate some early boot functions""
Now that <sys/tslog.h> is wrapped in #ifdef _KERNEL, it's safe to have
tslog annotations in files which might be built from userland (i.e. in
subr_boot.c, which is built as part of the boot loader).
Ben Harris [Fri, 7 Dec 2001 13:37:39 +0000 (13:37 +0000)]
Enable support for printing 8-byte integers. For some reason, most of the
code for this was present, but disabled.
This is required for POSIX compliance on platforms with 8-byte longs.
Colin Percival [Sun, 4 Jun 2023 18:39:45 +0000 (11:39 -0700)]
Revert "tslog: Annotate some early boot functions"
The change to subr_boot.c broke the libsa build because the TSLOG
macros have their own definitions for the boot loader -- I didn't
realize that the loader code used subr_boot.c.
I'm currently testing a fix and I'll revert this revert once I'm
satisfied that everything works, but I don't want to leave the
tree broken for too long.
Colin Percival [Tue, 30 May 2023 00:29:24 +0000 (17:29 -0700)]
tslog: Annotate parts of SYSINIT cpu
Booting an amd64 kernel on Firecracker with 1 CPU and 128 MB of RAM,
SYSINIT cpu takes roughly 2770 us:
* 2280 us in vm_ksubmap_init
* 535 us in kmem_malloc
* 450 us in pmap_zero_page
* 1720 us in pmap_growkernel
* 1620 us in pmap_zero_page
* 80 us in bufinit
* 480 us in cpu_setregs
* 430 us in cpu_setregs calling load_cr0
Much of this is hypervisor overhead: load_cr0 is slow because it traps
to the hypervisor, and 99% of the time in pmap_zero_page is spent when
we first touch the page, presumably due to the host Linux kernel
faulting in backing pages one by one.
Colin Percival [Mon, 29 May 2023 23:03:14 +0000 (16:03 -0700)]
tslog: Optionally instrument pmap_zero_page
Booting an amd64 kernel on Firecracker with 1 CPU and 128 MB of RAM,
pmap_zero_page is responsible for 4.6 ms of the 25.0 ms of boot time.
This is not in fact time spent zeroing pages though; almost all of
that time is spent in a first-touch penalty, presumably due to the
host Linux kernel faulting in backing pages one by one.
There's probably a way to improve that by teaching Firecracker to
fault in all the VM's pages from the start rather than having them
faulted in one at a time, but that's outside of FreeBSD's control.
This commit adds a TSLOG_PAGEZERO option which enables TSLOG on the
amd64 pmap_zero_page function; it's a separate option (turned off
by default even if TSLOG is enabled) since zeroing pages happens
enough that it can easily fill the TSLOG buffer and prevent other
timing information from being recorded.
Colin Percival [Mon, 22 May 2023 09:54:04 +0000 (02:54 -0700)]
tslog: Annotate some early boot functions
Booting an amd64 kernel on Firecracker with 1 CPU and 128 MB of RAM,
hammer_time takes roughly 2740 us:
* 55 us in xen_pvh_parse_preload_data
* 20 us in boot_parse_cmdline_delim
* 20 us in boot_env_to_howto
* 15 us in identify_hypervisor
* 1320 us in link_elf_reloc
* 1310 us in relocate_file1 handling ef->rela
* 25 us in init_param1
* 30 us in dpcpu_init
* 355 us in initializecpu
* 255 us in initializecpu calling load_cr4
* 425 us in getmemsize
* 280 us in pmap_bootstrap
* 205 us in create_pagetables
* 10 us in init_param2
* 25 us in pci_early_quirks
* 60 us in cninit
* 90 us in kdb_init
* 105 us in msgbufinit
* 20 us in fpuinit
* 205 us elsewhere in hammer_time
Some of these are unavoidable (e.g. identify_hypervisor uses CPUID and
load_cr4 loads the CR4 register, both of which trap to the hypervisor)
but others may deserve attention.
Colin Percival [Sun, 21 May 2023 20:03:33 +0000 (13:03 -0700)]
tslog: Handle curthread equal to NULL
Early in the kernel boot, curthread goes through three stages:
1. Kernel crash when you try to access it, because PCPU doesn't exist.
2. NULL, because PCU exists but isn't initialized.
3. &thread0, which is where most of the kernel boot process runs.
This broke TSLOG from inside hammer_time since the scripts which parse
logged records didn't understand that NULL meant &thread0.
Tell tslog to record &thread0 as the active thread if passed NULL.
Jamie Gritton [Sun, 4 Jun 2023 00:45:54 +0000 (17:45 -0700)]
jail: Allow nested jail definitions.
Make the jail.conf specification recursive, with jail definitions
allowed within a jail's parameter list, just as they are allowed
at the top level. Such inner jails are part of the outer jail's
hierarchy, as if they were specified with hierarchical names.
Daniel Kolesa [Sat, 3 Jun 2023 21:08:56 +0000 (23:08 +0200)]
sh(1): initialize smark to zero in main()
As popstackmark may be called on this without pushstackmark having
been called, we need to initialize it so that we don't get a bogus
comparison inside popstackmark, which would have resulted in a
NULL pointer dereference.
MFC After: 3 days
Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D40413
Centralize KINST_TRAMP_FILL_PATTERN and KINST_TRAMP_FILL_SIZE to reduce
redefinitions, and use the architecture-dependent kinst_patchval_t as
their size.
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40406
Doug Rabson [Fri, 2 Jun 2023 15:58:33 +0000 (16:58 +0100)]
pf: Fix tests broken by enabling inet-local filtering
Summary:
Three of the pf dummynet tests were using filter rules which matched
both the intended epair interface as well as lo0 which now receives
PFIL_OUT events for messages delivered to the local network stack (if
enabled). This commit changes the rules to match only for the expected
epair interface.
dd(1): neutralize SIGINT while non-async-signal safe code is executing
making the SIGINT handler (the terminate() function) safe to execute at
any interruption moment. This fixes a race in 5807f35c541c26bbd91a3ae12506cd8dd8f20688 where SIGINT delivered right
after the check_terminate() but before a blocking syscall would not
cause abort.
Do it by setting the in_io flag around potentially blocking io syscalls.
If handler sees the flag, it terminates the program. Otherwise,
termination is delegated to the before_io/after_io fences.
Reviewed by: Andrew Gierth <andrew@tao146.riddles.org.uk>
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D40281
Shailend Chand [Fri, 2 Jun 2023 18:58:24 +0000 (11:58 -0700)]
Add gve, the driver for Google Virtual NIC (gVNIC)
gVNIC is a virtual network interface designed specifically for
Google Compute Engine (GCE). It is required to support per-VM Tier_1
networking performance, and for using certain VM shapes on GCE.
The NIC supports TSO, Rx and Tx checksum offloads, and RSS.
It does not currently do hardware LRO, and thus the software-LRO
in the host is used instead. It also supports jumbo frames.
For each queue, the driver negotiates a set of pages with the NIC to
serve as a fixed bounce buffer, this precludes the use of iflib.
Mark Johnston [Fri, 2 Jun 2023 17:35:44 +0000 (13:35 -0400)]
ossl: Compile newly added files into the kernel if so requested
Fixes: 9a3444d91c70 ("ossl: Add a VAES-based AES-GCM implementation for amd64")
Fixes: 9b1d87286c78 ("ossl: Add a fallback AES-GCM implementation using AES-NI")
Mark Johnston [Fri, 2 Jun 2023 16:01:41 +0000 (12:01 -0400)]
ossl: Add a fallback AES-GCM implementation using AES-NI
This lets one use ossl(4) for AES-GCM operations on contemporary amd64
platforms. A kernel benchmark indicates that this gives roughly
equivalent throughput to aesni(4) for various buffer sizes.
Bulk processing is done in aesni-gcm-x86_64.S, the rest is handled in a
C wrapper ported from OpenSSL's gcm128.c.
Mark Johnston [Fri, 2 Jun 2023 15:58:29 +0000 (11:58 -0400)]
ossl: Add a VAES-based AES-GCM implementation for amd64
aes-gcm-avx512.S is generated from OpenSSL 3.1 and implements AES-GCM.
ossl_x86.c detects whether the CPU implements the required AVX512
instructions; if not, the ossl(4) module does not provide an AES-GCM
implementation. The VAES implementation increases throughput for all
buffer sizes in both directions, up to 2x for sufficiently large
buffers.
The "process" implementation is in two parts: a generic OCF layer in
ossl_aes.c that calls a set of MD functions to do the heavy lifting.
The intent there is to make it possible to add other implementations for
other platforms, e.g., to reduce the diff required for D37421.
A follow-up commit will add a fallback path to legacy AES-NI, so that
ossl(4) can be used in preference to aesni(4) on all amd64 platforms.
In the long term we would like to replace aesni(4) and armv8crypto(4)
with ossl(4).
Note, currently this implementation will not be selected by default
since aesni(4) and ossl(4) return the same probe priority for crypto
sessions, and the opencrypto framework selects the first registered
implementation to break a tie. Since aesni(4) is compiled into the
kernel, aesni(4) wins. A separate change may modify ossl(4) to have
priority.
Mark Johnston [Fri, 2 Jun 2023 15:57:38 +0000 (11:57 -0400)]
ossl: Expose more CPUID bits in OPENSSL_ia32cap_P
This is needed to let OpenSSL 3.1 routines detect VAES and VPCLMULQDQ
extensions. The intent is to import ASM routines which implement
AES-GCM using VEX-prefixed AES-NI instructions.
Kristof Provost [Tue, 30 May 2023 19:17:54 +0000 (21:17 +0200)]
pf: carry over rule actions from route-to rules
If we route-to (or dup-to/reply-to) we re-run pf_test(), which will also
create states for the connection.
This means that we may end up matching a different (i.e. not the state
that was created by the route-to rule) state, without the attributes
(such as dummynet pipes/queues) set by the route-to rule.
Address this by inheriting the pf_rule_actions from the route-to rule
while evaluating the connection again in pf_test(). That is, we set
default pf_rule_actions based on the route-to rule for the new
evaluation. The new rule may still overrule these, but if it does not
have such actions the route-to actions are applied.
Do the same for IPv6 rules in pf_test6()/pf_route6().
Pierre Pronchery [Thu, 25 May 2023 17:09:27 +0000 (19:09 +0200)]
dumpon: Request the OpenSSL 1.1 API
OPENSSL_API_COMPAT can be used to specify the OpenSSL API version in
use for the purpose of hiding deprecated interfaces and enabling
the appropriate deprecation notices.
This change is a NFC while we're still using OpenSSL 1.1.1 but will
avoid deprecation warnings upon the switch to OpenSSL 3.0.
A future update may migrate to use the OpenSSL 3.0 APIs.
PR: 271615
Pull request: https://github.com/freebsd/freebsd-src/pull/757
Sponsored by: The FreeBSD Foundation
projects / FreeBSD / FreeBSD.git / log