kp [Tue, 23 Jan 2018 04:30:46 +0000 (04:30 +0000)]
MFC r327675
pf: Avoid integer overflow issues by using mallocarray() iso. malloc()
pfioctl() handles several ioctl that takes variable length input, these
include:
- DIOCRADDTABLES
- DIOCRDELTABLES
- DIOCRGETTABLES
- DIOCRGETTSTATS
- DIOCRCLRTSTATS
- DIOCRSETTFLAGS
All of them take a pfioc_table struct as input from userland. One of
its elements (pfrio_size) is used in a buffer length calculation.
The calculation contains an integer overflow which if triggered can lead
to out of bound reads and writes later on.
Reported by: Ilja Van Sprundel <ivansprundel@ioactive.com>
cy [Tue, 23 Jan 2018 04:01:48 +0000 (04:01 +0000)]
MFC r327718:
When growing the state, also grow the seed array. Otherwise memory
that was not allocated will be accessed.
This necessitated refactoring state seed allocation from
ipf_state_soft_init() into a new common ipf_state_seed_alloc() function
as it is now also used by ipf_state_rehash() when changing the size of
the state hash table in addition to by ipf_state_soft_init() during
initialization.
According to Christos Zoulas <christos@NetBSD.org>:
The bug was encountered by a NetBSD vendor who's customer machines had
large ipfilter states. The bug was reliably triggered by resizing the
state variables using "ipf -T".
https://www.illumos.org/issues/8373
The code that writes ZIL blocks uses dmu_tx_assign(TXG_WAIT) to assign
a transaction to a transaction group. That seems to be logically
incorrect as writing of the ZIL block does not introduce any new dirty
data. Also, when there is a lot of dirty data, the call can introduce
significant delays into the ZIL commit path, thus affecting all
synchronous writes. Additionally, ARC throttling may affect the ZIL
writing.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Approved by: Dan McDonald <danmcd@joyent.com>
Author: Andriy Gapon <avg@FreeBSD.org>
kp [Sat, 20 Jan 2018 23:46:03 +0000 (23:46 +0000)]
MFC r327674, r327796
Introduce mallocarray() in the kernel
Similar to calloc() the mallocarray() function checks for integer
overflows before allocating memory.
It does not zero memory, unless the M_ZERO flag is set.
Additionally, move the overflow check logic out to WOULD_OVERFLOW() for
consumers to have a common means of testing for overflowing allocations.
WOULD_OVERFLOW() should be a secondary check -- on 64-bit platforms, just
because an allocation won't overflow size_t does not mean it is a sane size
to request. Callers should be imposing reasonable allocation limits far,
far, below overflow.
cy [Sat, 20 Jan 2018 18:00:42 +0000 (18:00 +0000)]
MFC r327912:
Though this block of code is not used by FreeBSD, correct a call to
sprintf() with a macro call to SNPRINTF similar to other calls to
SNPRINTF within this same block.
mav [Fri, 19 Jan 2018 16:06:52 +0000 (16:06 +0000)]
MFC r323317 (by scottl): Move the intrhook release to later in the function
so that GEOM knows to wait longer for possible root devices to come online.
This fixes a race that seems to be triggered by EARLY_AP_STARTUP.
kevans [Thu, 18 Jan 2018 22:13:31 +0000 (22:13 +0000)]
MFC r322288: regex(3): Refactor fast/slow stepping bits in matching engine
Adding features for matching is fairly straightforward, but this requires
some duplication because of this fast/slow setup. They can be fairly
trivially combined into a single walk(), so do it to make future additions
less error prone.
kevans [Thu, 18 Jan 2018 22:10:00 +0000 (22:10 +0000)]
MFC r320742, r320750, r320796: Refactor regex(3) for maintainability
MFC r320742:
The impending libregex will implement GNU extensions to bring BREs and
EREs closer together. Prepare for this and reduce the diff of libregex
changes by refactoring and combining the top-level parsers for EREs/BREs
ahead of time.
Branching functionality has been split out to make it easier to follow the
combined version of the top-level parser. It may also be enabled in the
parsing context to make it easier when libregex enables branching for BREs.
A branching context was also added for the various branching functions and
so that BREs, for instance, can determine if they're the first expression in
a chain of expressions within the current branch and treat '*' as ordinary
if so.
This should have no functional impact and negligible performance impact.
MFC r320750: Fix sparc64 libc build after r320742.
p_branch_empty was declared but never used due to an oversight. Use it as
designed, further comment on its return value.
MFC r320796:
Correctly ignore branch operators in the top-level parser when applicable.
An oversight in r320742 caused BREs to become sensitive to the branching
operator prematurely, which caused breakage in some limited situations --
namely, those that tried to use branching in a BRE. Most of these scenarios
had already been corrected beforehand to properly use gsed or grep for GNU
extensions, so the damage is slightly mitigated.
kevans [Thu, 18 Jan 2018 21:59:13 +0000 (21:59 +0000)]
MFC r327567: hexdump(1): Speed up -s flag on devices
Using the -s flag on devices is extraordinarily slow due to using fseek(3) a
little too conservatively. Address this by using fseek on character/block
devices as well, falling back to getchar(3) only if we fail to seek or we're
operating on tape drives, where fseek may succeed while not actually being
supported.
kevans [Thu, 18 Jan 2018 21:53:07 +0000 (21:53 +0000)]
MFC r326084: patch(1): don't assume match if we run out of context to check
Patches with very little context (-U0 and -U1) could get misapplied if
the file to be patched changes and a hunk is no longer applicable. Matching
with fuzz would be attempted and default to a match when we unexpectedly ran
out of context.
This also affected patches with higher levels of context but had limited
actual context due to the hunk being located near the beginning/end of file.
kevans [Thu, 18 Jan 2018 21:46:42 +0000 (21:46 +0000)]
MFC r324431: patch(1): Don't overrun line buffer in some cases
Patches like file.txt attached to PR 190195 with a final line formed
like ">(EOL)" could cause a copy past the end of the current line buffer. In
the case of PR 191641, this caused a duplicate line to be copied into the
resulting file.
Instead of running past the end, treat it as if it were a blank line.
kevans [Thu, 18 Jan 2018 21:39:03 +0000 (21:39 +0000)]
MFC r322322: Expose device caps as libusb_bos_descriptor::dev_capability
Some libusb consumers in Linux-land (in this case, libusb4java) expect a
dev_capability member that they can use to enumerate the device
capabilities.
No particular layout is expected of this, just that it can be traversed
using the bLength member until bNumDeviceCapabilities are read and that the
consumer may then use one of the libusb_get_*_descriptor methods to extract
specific (usb 2.0 vs. ss) capability information.
kevans [Thu, 18 Jan 2018 21:29:46 +0000 (21:29 +0000)]
MFC r322177: Respect SIMPLE_BACKUP_SUFFIX environment variable in indent(1)
Instead of using a non-configurable ".BAK" suffix, respect the
SIMPLE_BACKUP_SUFFIX environment variable also used by patch(1). This
simplifies cleanup operations in some patch/indent workflows.
ed [Thu, 18 Jan 2018 13:43:09 +0000 (13:43 +0000)]
MFC r327560:
Allow timed waits with relative timeouts on locks and condvars.
Even though pthreads doesn't support this, there are various alternative
APIs that use this. For example, uv_cond_timedwait() accepts a relative
timeout. So does Rust's std::sync::Condvar::wait_timeout().
Though I personally think that relative timeouts are bad (due to
imprecision for repeated operations), it does seem that people want
this. Extend the existing futex functions to keep track of whether an
absolute timeout is used in a boolean flag.
Pull in r314838 from upstream clang trunk (by Richard Smith):
Suppress -Wmissing-braces warning when aggregate-initializing a
struct with a single field that is itself an aggregate.
In C++, such initialization of std::array<T, N> types is guaranteed
to work by the standard, is completely idiomatic, and the "suggested"
alternative from Clang was technically invalid.
Together, these suppress unneeded "suggest braces around initialization
of subobject" warnings for C++11 initializer lists.
kevans [Sun, 14 Jan 2018 05:10:02 +0000 (05:10 +0000)]
MFC (conceptually) r327377: Pull in strnlen for sys/boot environment
This is a direct commit to stable/11 due to restructuring of lib/libstand =>
stand/libsa in -HEAD. The diff remains the same and it is simply applied to
the previous location.
MFC r327377: Pull in strnlen from libc
strnlen is not used at the moment, but it will be when libfdt gets updated.
Prepare for the not-so-distant future by pulling in strnlen.
This is a direct commit to stable/11 due to restructuring of sys/boot =>
stand in -HEAD. The diff remains the same and it is simply applied to the
previous location.
MFC r327376: stand/fdt: Swap libfdt include order
libfdt.h should be included before fdt.h, as hinted at by all of libfdt/;
standard include order being libfdt.h, libfdt_env.h, fdt.h.
The current include order also causes problems when libfdt gets updated, as
fdt.h requires some definitions from libfdt.h.
MFC r327416: stand/fdt: Make fdt_overlay_apply signature-compatible with
libfdt
libfdt will assume a writable fdt overlay blob has been passed in, so make
ours compatible to allow easier review when we try to drop libfdt into
place. overlay from the calling context is writable, making it safe to
simply rip out everything related to copying the overlay blob in
fdt_overlay_apply.
I note here that we still have problems: fdt_overlay_apply, both our version
and libfdt's, may fail and have already clobbered the base fdt to some
extent. Future work will make sure we don't apply a potentially bogus fdt,
instead discarding the base fdt if we had an error.
This is a direct commit to stable/11 due to restructuring of sys/boot =>
stand in -HEAD. The diff remains the same and it is simply applied to the
previous location.
This is a direct commit to stable/11 due to restructuring of sys/boot =>
stand in -HEAD. The diff remains the same and it simply applied to the
previous location.
MFC r327298: stand/fdt: Fix loading of multiple fdt_overlays
fdt_load_dtb_overlays was written to unload previous overlay when a new
valid one is come across. fdt_apply_overlays further down is written to
iterate over all .dtbo's currently loaded and apply them one-by-one. Correct
fdt_load_dtb_overlays to stop dropping valid overlays that were previously
loaded and match expectations.
MFC r327299: stand/fdt: Avoid bailout when dtbo has no fixups
In the case of a simple dtbo where fragment uses target-path and the overlay
contains no references, /__fixups__ will not be included by either our dtc
or dtc from ports, but the file still has valid fragments to be applied.
Additional testing found that /__symbols__ might also be omitted if it's
empty, which is not necessarily an error.
karels [Sat, 13 Jan 2018 14:10:05 +0000 (14:10 +0000)]
MFC r327505:
make SW_WATCHDOG dynamic
Enable the hardclock-based watchdog previously conditional on the
SW_WATCHDOG option whenever hardware watchdogs are not found, and
watchdogd attempts to enable the watchdog. The SW_WATCHDOG option
still causes the sofware watchdog to be enabled even if there is a
hardware watchdog. This does not change the other software-based
watchdog enabled by the --softtimeout option to watchdogd.
Note that the code to reprime the watchdog during kernel core dumps is
no longer conditional on SW_WATCHDOG. I think this was previously a bug.
eadler [Sat, 13 Jan 2018 09:28:43 +0000 (09:28 +0000)]
MFC r324206:
bsdinstall(8) hardening menu: Utilize new kern.randompid=1 behaviour
Enabling the PID randomization option in bsdinstall(8)'s hardening menu
now randomizes the effective value of kern.randompid on each boot.
Previous behaviour:
When kern.randompid was enabled via the the bsdinstall(8) hardening menu,
a random value was generated and placed in the systems /etc/sysctl.conf as
kern.randompid=value
This makes the value of kern.randompid static across reboots.
New behaviour:
When kern.randompid is enabled via the bsdinstall(8) hardening menu, the
line kern.randompid=1 is placed in the systems /etc/sysctl.conf.
This takes advantage of a new kernel feature and makes the value of
kern.randompid be randomized by the kernel on each reboot.
pfg [Fri, 12 Jan 2018 17:36:19 +0000 (17:36 +0000)]
MFC r327697, r327699:
Use the __result_use_check attribute for reallocf(9).
MFC r327751:
Use the __result_use_check attribute also for reallocf(3).
The GCC attribute causes a warning to be emitted if a caller of the
function with this attribute does not use its return value. Unlike the
traditional realloc, with reallocf(3) we don't have to check for NULL
values but we still have to make sure the result is used.
cy [Fri, 12 Jan 2018 02:49:18 +0000 (02:49 +0000)]
MFC 327737:
USNO and possibly others have misinterpreted the maining of the
leapseconds last-update field and incorrectly increment it when changing
the file even though the leapsecond data has not changed. For instance,
if a leapsecond file is obtained from USNO, when it expires it will not
be replaced by a newer file from other sources because it has an
incorrect later last-update (version).
asomers [Fri, 12 Jan 2018 00:31:07 +0000 (00:31 +0000)]
MFC r313962, r313972-r313973, r315230
r313962 by allanjude:
improve PBKDF2 performance
The PBKDF2 in sys/geom/eli/pkcs5v2.c is around half the speed it could be
GELI's PBKDF2 uses a simple benchmark to determine a number of iterations
that will takes approximately 2 seconds. The security provided is actually
half what is expected, because an attacker could use the optimized
algorithm to brute force the key in half the expected time.
With this change, all newly generated GELI keys will be approximately 2x
as strong. Previously generated keys will talk half as long to calculate,
resulting in faster mounting of encrypted volumes. Users may choose to
rekey, to generate a new key with the larger default number of iterations
using the geli(8) setkey command.
Security of existing data is not compromised, as ~1 second per brute force
attempt is still a very high threshold.
r313973 by ngie:
A forced commit to note other portion of the Makefile change accidentally
committed in r313972
The code committed in r313962 implicitly relies on python 2.x to generate
testvect.h . There are a handful of issues with this approach:
- python is not an explicit build dependency for FreeBSD
- python 2.x is deprecated and will be removed sometime in the future
(potentially before 11.x's EOL), and the script does not function with
python 3.5 (it uses deprecated idioms and incompatible function calls).
- python(1) (by default) lives in /usr/local/bin (${LOCALBASE}/bin) and
gentestvect.py is a dependency of testvect.h (prior to r313972) which
means that if the mtime of the generator script was newer than the
mtime of the test vector, it could cause a spurious build failure in
build time or at install time.
r315230 by ngie:
Move .../sys/geom/eli/pbkdf2... to .../sys/geom/class/eli/...
This change moves the tests added in r313962 to an existing directory
structure used by the geli TAP tests. It also, renames the test from
pbkdf2 to pbkdf2_test .
The changes to ObsoleteFiles.inc are being committed separately as they
aren't needed for the MFC to ^/stable/11, etc, if the MFC for the tests
is done all in one commit.
asomers [Fri, 12 Jan 2018 00:25:56 +0000 (00:25 +0000)]
MFC r309373 (by bdrewery)
Fix setrlimit_test:setrlimit_memlock when the system has exceeded vm.max_wired.
This uses the same fix as r294894 did for the mlock test. The code from
that commit is moved into a common object file which PROGS supports
building first.
asomers [Thu, 11 Jan 2018 23:57:55 +0000 (23:57 +0000)]
MFC r326640:
Optimize telldir(3)
Currently each call to telldir() requires a malloc and adds an entry to a
linked list which must be traversed on future telldir(), seekdir(),
closedir(), and readdir() calls. Applications that call telldir() for every
directory entry incur O(n^2) behavior in readdir() and O(n) in telldir() and
closedir().
This optimization eliminates the malloc() and linked list in most cases by
packing the relevant information into a single long. On 64-bit architectures
msdosfs, NFS, tmpfs, UFS, and ZFS can all use the packed representation. On
32-bit architectures msdosfs, NFS, and UFS can use the packed
representation, but ZFS and tmpfs can only use it for about the first 128
files per directory. Memory savings is about 50 bytes per telldir(3) call.
Speedup for telldir()-heavy directory traversals is about 20-30x for one
million files per directory.
asomers [Thu, 11 Jan 2018 23:56:55 +0000 (23:56 +0000)]
MFC r326624:
cam: fix sign-extension error in adagetparams
adagetparams contains a sign-extension error that will cause the sector
count to be incorrectly calculated for ATA disks of >=1TiB that still use
CHS addressing. Disks using LBA48 addressing are unaffected.
ian [Thu, 11 Jan 2018 17:51:22 +0000 (17:51 +0000)]
Do not build lint(1) by default on stable-11, add WITH_LINT to enable
building it.
lint(1) is required on the host build system to build lint(1) libraries, and
it is no longer available on FreeBSD 12. This prevents using a 12 or later
host to build an 11 or earlier system, which causes problems for building
jails and using poudriere. The problem could be fixed by treating lint as a
bootstrap tool when building on 12+, but that just adds complexity and build
time to build a broken tool that virtually nobody uses anymore.
This is a direct commit to 11-stable because lint no longer exists in later
branches.
pfg [Wed, 10 Jan 2018 21:17:33 +0000 (21:17 +0000)]
MFC r327289:
rpc.sprayd: Bring some changes from NetBSD.
Most notable, other than some style issues:
CVS 1.11:
do not use LOG_CONS.
CVS 1.13:
consistently use exit instead of return in main().
use LOG_WARNING instead of LOG_ERR for non critical errors.
emaste [Tue, 9 Jan 2018 00:27:02 +0000 (00:27 +0000)]
MFC r327489: elfcopy: copy raw (untranslated) contents to binary output
Previously elfcopy used elf_getdata to obtain data from ELF sections
being copied to binary output, but elf_getdata returns data that has
been translated - that is, data is in host byte order. When the host and
target differ in endianness (e.g., converting a big-endian MIPS ELF
object to binary on an x86 host) this resulted in byte-swapped data in
certain sections such as .dynamic.
Instead use elf_rawdata to keep data in the original, target endianness.
Reported by: Hiroki Mori <yamori83@yahoo.co.jp>, Bill Yuan
Sponsored by: The FreeBSD Foundation
bapt [Sun, 7 Jan 2018 10:29:15 +0000 (10:29 +0000)]
MFC r326769:
Remove hard coded number of lun definition
The number of lun exposed is now exposed via sysctl by the kernel.
Use that number in ctlstat instead of the hardcoded version
Add a backward compatibility in case the sysctl(2) request fails.
This also allows ctlstat -l 1118 to actually work when having more than
1024 luns.
Reviewed by: avg, manu (both before the backward compatibility addition)
Approved by: avg, manu (both before the backward compatibility addition)
Sponsored by: Gandi.net
Differential Revision: https://reviews.freebsd.org/D13446
ian [Sun, 7 Jan 2018 02:59:27 +0000 (02:59 +0000)]
MFC r327220-r327221
r327220:
Update the FDT example for the lm75 sensor to match current devicetree
standards and what the existing driver expects.
Also change 'like' to 'such as' where the text is providing an example
rather than a simile.
r327221:
Complete the changing of the old "i2c-address" property to the modern "reg"
property by updating the description text to match the updated example.
ian [Sun, 7 Jan 2018 02:57:35 +0000 (02:57 +0000)]
MFC r327222:
Add a new ARM kernel option, LOCORE_MAP_MB, to control the size of the
kernel VA mapping in the temporary page tables set up by locore-v6.S.
The number used to be hard-coded to 64MB, which is still the default if
the kernel option is not specified. However, 64MB is insufficient for
using a large mdroot filesystem. The hard-coded number can't be safely
increased because too large a number may run into memory-mapped IO space
on some SoCs that must not be mapped as ordinary memory.
ian [Sun, 7 Jan 2018 01:08:10 +0000 (01:08 +0000)]
MFC r327226, r327356
r327226:
Add a section describing how to tune ARM kernel options to use an MD_ROOT
filesystem larger than about 50-55 MiB.
The description of VM_KMEM_SIZE_SCALE is roughly as hand-wavy as my
understanding of the option, but at least mentioning that it's a factor
and giving an empirical datapoint that works will give folks some idea
of what to tweak if they have problems.
r327356:
Correct a mistake and reword a couple sentences to clarify that "the value"
refers to the scale value, not the kmem_arena size that results from scaling.
ian [Sun, 7 Jan 2018 01:06:04 +0000 (01:06 +0000)]
MFC r327439:
Allow use of .WAIT in the LOCAL_DIRS and LOCAL_LIB_DIRS lists.
A comment in Makefile.inc1 has long stated that LOCAL_DIRS are built last,
after the base system. Incremental improvements in parallel building over
the years have led to LOCAL_DIRS being built in parallel with base system
directories. This change allows the .WAIT directive to appear in LOCAL_DIRS
and LOCAL_LIB_DIRS lists to give the user some control over parallel
building of local additions.