MFC: r320062, r320070, r320126
This is a partial merge of only the NFS changes and not the maxbcachebuf
tunable.
The NFS client changes make the code handle different I/O sizes more
correctly. However, with the limit at 64K, they are not actually
necessary.
This MFC is mainly being done so that subsequent MFCs to the NFS
code will merge easily.
MFC r319699
When allocating swap blocks, if the available number of free blocks in a
subtree is already zero, then setting the "largest contiguous free block"
hint for that subtree to anything other than zero makes no sense. (To be
clear, assigning a value to the hint that is too large is not a correctness
problem, only a pessimization.)
MFC r319755
blist_fill()'s return type is too narrow. blist_fill() accepts a 64-bit
quantity as the size of the range to fill, but returns a 32-bit quantity
as the number of blocks that were allocated to fill that range. This
revision corrects that mismatch.
MFC r319793
Remove an unnecessary field from struct blist. (The comment describing
what this field represented was also inaccurate.)
In r178792, blist_create() grew a malloc flag, allowing M_NOWAIT to be
specified. However, blist_create() was not modified to handle the
possibility that a malloc() call failed. Address this omission.
Increase the width of the local variable "radix" to 64 bits. This
matches the width of the corresponding field in struct blist.
ken [Mon, 3 Jul 2017 15:34:19 +0000 (15:34 +0000)]
MFC r320421:
------------------------------------------------------------------------
r320421 | ken | 2017-06-27 13:26:02 -0600 (Tue, 27 Jun 2017) | 37 lines
Fix a panic in camperiphfree().
If a peripheral driver (e.g. da, sa, cd) is added or removed from the
peripheral driver list while an unrelated peripheral driver instance (e.g.
da0, sa5, cd2) is going away and is inside camperiphfree(), we could
dereference an invalid pointer.
When peripheral drivers are added or removed (see periphdriver_register()
and periphdriver_unregister()), the peripheral driver array is resized
and existing entries are moved.
Although we hold the topology lock while we traverse the peripheral driver
list, we retain a pointer to the location of the peripheral driver pointer
and then drop the topology lock. So we are still vulnerable to the list
getting moved around while the lock is dropped.
To solve the problem, cache a copy of the peripheral driver pointer. If
its storage location in the list changes while we have the lock dropped, it
won't have any effect.
This doesn't solve the issue that peripheral drivers ("da", "cd", as opposed
to individual instances like "da0", "cd0") are not generally part of a
reference counting scheme to guard against deregistering them while there
are instances active. The caller (generally the person unloading a module)
has to be aware of active drivers and not unload something that is in use.
sys/cam/cam_periph.c:
In camperiphfree(), cache a pointer to the peripheral driver
instance to avoid holding a pointer to an invalid memory location
in the event that the peripheral driver list changes while we have
the topology lock dropped.
MFC r320049
Pages that are passed to swap_pager_putpages() should already be fully
dirty. Assert that they are fully dirty rather than redundantly calling
vm_page_dirty() on them.
jilles [Fri, 30 Jun 2017 21:32:48 +0000 (21:32 +0000)]
MFC r315005: sh: Fix executing wrong command with ${x#$(y)}$(z).
The parsed internal representation of words consists of a byte string with a
list of nodes (commands in command substitution). Each unescaped CTLBACKQ or
CTLBACKQ | CTLQUOTE byte corresponds to an entry in the list.
If param in ${param#%##%%word} is not set, the word is not expanded (in a
deviation of POSIX shared with other ash variants and ksh93). Erroneously,
the pointer in the list of commands (argbackq) was not advanced. This caused
the wrong command to be executed later if the outer word contained another
command substitution.
Example:
echo "${unsetvar#$(echo a)}$(echo b)"
wrote "a" but should write "b".
avg [Fri, 30 Jun 2017 09:34:50 +0000 (09:34 +0000)]
MFC r308782:
After some ZIL changes 6 years ago zil_slog_limit got partially broken
due to zl_itx_list_sz not updated when async itx'es upgraded to sync.
Actually because of other changes about that time zl_itx_list_sz is not
really required to implement the functionality, so this patch removes
some unneeded broken code and variables.
Original idea of zil_slog_limit was to reduce chance of SLOG abuse by
single heavy logger, that increased latency for other (more latency critical)
loggers, by pushing heavy log out into the main pool instead of SLOG. Beside
huge latency increase for heavy writers, this implementation caused double
write of all data, since the log records were explicitly prepared for SLOG.
Since we now have I/O scheduler, I've found it can be much more efficient
to reduce priority of heavy logger SLOG writes from ZIO_PRIORITY_SYNC_WRITE
to ZIO_PRIORITY_ASYNC_WRITE, while still leave them on SLOG.
Existing ZIL implementation had problem with space efficiency when it
has to write large chunks of data into log blocks of limited size. In some
cases efficiency stopped to almost as low as 50%. In case of ZIL stored on
spinning rust, that also reduced log write speed in half, since head had to
uselessly fly over allocated but not written areas. This change improves
the situation by offloading problematic operations from z*_log_write() to
zil_lwb_commit(), which knows real situation of log blocks allocation and
can split large requests into pieces much more efficiently. Also as side
effect it removes one of two data copy operations done by ZIL code WR_COPIED
case.
While there, untangle and unify code of z*_log_write() functions.
Also zfs_log_write() alike to zvol_log_write() can now handle writes crossing
block boundary, that may also improve efficiency if ZPL is made to do that.
alc [Wed, 28 Jun 2017 04:23:20 +0000 (04:23 +0000)]
MFC r314310
Refine the fix from r312954. Specifically, add a new PDE-only flag,
PG_PROMOTED, that indicates whether lingering 4KB page mappings might
need to be flushed on a PDE change that restricts or destroys a 2MB
page mapping. This flag allows the pmap to avoid range invalidations
that are both unnecessary and costly.
ken [Mon, 26 Jun 2017 15:23:12 +0000 (15:23 +0000)]
MFC r320123:
Fix a potential sleep while holding a mutex in the sa(4) driver.
If the user issues a MTIOCEXTGET ioctl, and the tape drive in question has
a serial number that is longer than 80 characters, we malloc a buffer in
saextget() to hold the output of cam_strvis().
Since a mutex is held in that codepath, doing a M_WAITOK malloc could lead
to sleeping while holding a mutex. Change it to a M_NOWAIT malloc and bail
out if we fail to allocate the memory. Devices with serial numbers longer
than 80 bytes are very rare (I don't recall seeing one), so this
should be a very unusual case to hit. But it is a bug that should be fixed.
sys/cam/scsi/scsi_sa.c:
In saextget(), if we need to malloc a buffer to hold the output of
cam_strvis(), don't wait for the memory. Fail and return an error
if we can't allocate the memory immediately.
As per https://datacenter.iers.org/eop/-/somos/5Rgv/latest/16:
INTERNATIONAL EARTH ROTATION AND REFERENCE SYSTEMS SERVICE (IERS)
SERVICE INTERNATIONAL DE LA ROTATION TERRESTRE ET DES SYSTEMES DE REFERENCE
SERVICE DE LA ROTATION TERRESTRE
OBSERVATOIRE DE PARIS
61, Av. de l'Observatoire 75014 PARIS (France)
Tel. : 33 (0) 1 40 51 23 35
FAX : 33 (0) 1 40 51 22 91
Internet : services.iers@obspm.fr
Paris, 9 January 2017
Bulletin C 53
To authorities responsible
for the measurement and
distribution of time
INFORMATION ON UTC - TAI
NO leap second will be introduced at the end of June 2017.
The difference between Coordinated Universal Time UTC and the
International Atomic Time TAI is :
from 2017 January 1, 0h UTC, until further notice : UTC-TAI = -37 s
Leap seconds can be introduced in UTC at the end of the months of December
or June, depending on the evolution of UT1-TAI. Bulletin C is mailed every
six months, either to announce a time step in UTC, or to confirm that there
will be no time step at the next possible date.
Christian BIZOUARD
Director
Earth Orientation Center of IERS
Observatoire de Paris, France
sephe [Mon, 26 Jun 2017 02:33:59 +0000 (02:33 +0000)]
MFC 320184
hyperv/storvsc: Reduce log verbosity
On some windows hosts TEST_UNIT_READY command will return
SRB_STATUS_ERROR and sense data "NOT READY asc:3a,1 (Medium
not present - tray closed)", this occurs periodically, and
not hurt anything else. So, we prefer to ignore this kind
of errors.
PR: 219973
Submitted by: Hongjiang Zhang <hongzhan microsoft com>
Sponsored by: Microsoft
Differential Revision: https://reviews.freebsd.org/D11271
delphij [Sun, 25 Jun 2017 05:46:03 +0000 (05:46 +0000)]
MFC r320216: Fix use-after-free introduced in r300388.
In r300388, endnetconfig() was called on nc_handle which would release
the associated netconfig structure, which means tmpnconf->nc_netid
would be a use-after-free.
Solve this by doing endnetconfig() in return paths instead.
ngie [Thu, 22 Jun 2017 07:42:36 +0000 (07:42 +0000)]
MFC r318545:
Install {cron.d,newsyslog.conf.d,syslog.d} via `make distribution`, not `make install`
I incorrectly started this pattern in r277541 with the opensm newsyslog.conf.d file,
and continued using it in r318441 and r318443.
This will fix the files being handled improperly via installworld, preventing tools like
etcupdate, mergemaster, etc from functioning properly when comparing the installed
contents on a system vs the contents in a source tree when doing merges.
Conditionally handle the crontab entry for atrun(8)
The default crontab prior to this commit assumes atrun(8) is always
present, which isn't true if MK_AT == no. Move atrun(8) execution
from /etc/crontab to /etc/cron.d/at, and base /etc/cron.d/at's installation
on MK_AT. cron(8) will detect /etc/cron.d/at's presence when the configuration
is loaded and run atrun every 5 minutes like it would prior to this commit.
SHELL and PATH are duplicated between /etc/crontab and /etc/cron.d/at
because atrun(8) executes programs, which may rely on environment
set in the current default /etc/crontab.
Noted by: bdrewery (in an internal review)
Relnotes: yes (may need to add environmental modifications to
/etc/cron.d/at)
jhb [Wed, 21 Jun 2017 14:39:31 +0000 (14:39 +0000)]
MFC 313186, 319702: Account for overhead of page structures when sizing page array.
313186:
Over the years, the code and comments in vm_page_startup() have diverged in
one respect. When determining how many page structures to allocate,
contrary to what the comments say, the code does not account for the
overhead of a page structure per page of physical memory. This revision
changes the code to match the comments.
319702:
Fix an off-by-one error in the VM page array on some systems.
r313186 changed how the size of the VM page array was calculated to be
less wasteful. For most systems, the amount of memory is divided by
the overhead required by each page (a page of data plus a struct vm_page)
to determine the maximum number of available pages. However, if the
remainder for the first non-available page was at least a page of data
(so that the only memory missing was a struct vm_page), this last page
was left in phys_avail[] but was not allocated an entry in the VM page
array. Handle this case by explicitly excluding the page from
phys_avail[].
marius [Tue, 20 Jun 2017 19:59:57 +0000 (19:59 +0000)]
The GEOM KPI in stable/10 requires Giant to be unacquired (a requirement
dropped in r300287). So wrap calling into GEOM in {DROP,PICKUP}_GIANT(),
similar to what as been done prio to r300288. This fixes a "Giant owned"
panic with r318159 in place and INVARIANTS enabled.
Reported by: Oliver Pinter
davidcs [Tue, 20 Jun 2017 18:52:35 +0000 (18:52 +0000)]
MFC r319964
Upgrade STORMFW to 8.30.0.0 and ecore version to 8.30.0.0
Add support for pci deviceID 0x8070 for QLE41xxx product line which
supports 10GbE/25GbE/40GbE
jhb [Tue, 20 Jun 2017 15:55:15 +0000 (15:55 +0000)]
MFC 319490: Remove stale cap_rights_get(2) manpage.
The documentation moved to section 3 several years ago, but
'man cap_rights_get' pulls up cap_rights_limit(2) (which is
MLINKed to cap_rights_get.2) instead of cap_rights_get(3).
hselasky [Sun, 18 Jun 2017 11:48:40 +0000 (11:48 +0000)]
MFC r319972:
Use static device numbering instead of dynamic one when creating
mlx4en network interfaces. This prevents infinite unit number growth
typically when the mlx4en driver is used inside virtual machines which
support runtime PCI attach and detach.
delphij [Thu, 15 Jun 2017 04:37:23 +0000 (04:37 +0000)]
MFC r319852:
Fix buffer lengths.
After r319369, the RPC code validates caller supplied buffer length in
taddr2uaddr. When no -h is specified, the sizeof(ai_addr) is used,
which is always smaller than the required size and therefore uaddr
would be NULL, causing the kernel to copyin() from userland NULL
and fail with EFAULT.
alc [Thu, 15 Jun 2017 03:58:23 +0000 (03:58 +0000)]
MFC r318995
In r118390, the swap pager's approach to striping swap allocation over
multiple devices was changed. However, swapoff_one() was not fully and
correctly converted. In particular, with r118390's introduction of a per-
device blist, the maximum swap block size, "dmmax", became irrelevant to
swapoff_one()'s operation. Moreover, swapoff_one() was performing out-of-
range operations on the per-device blist that were silently ignored by
blist_fill().
This change corrects both of these problems with swapoff_one(), which will
allow us to potentially increase MAX_PAGEOUT_CLUSTER. Previously,
swapoff_one() would panic inside of blist_fill() if you increased
MAX_PAGEOUT_CLUSTER.
MFC r319001
After r118390, the variable "dmmax" was neither the correct strip size
nor the correct maximum block size. Moreover, after r318995, it serves
no purpose except to provide information to user space through a read-
sysctl.
This change eliminates the variable "dmmax" but retains the sysctl. It
also corrects the value returned by the sysctl.
MFC r319604
Halve the memory being internally allocated by the blist allocator. In
short, half of the memory that is allocated to implement the radix tree is
wasted because we did not change "u_daddr_t" to be a 64-bit unsigned int
when we changed "daddr_t" to be a 64-bit (signed) int. (See r96849 and
r96851.)
MFC r319612
When the function blist_fill() was added to the kernel in r107913, the swap
pager used a different scheme for striping the allocation of swap space
across multiple devices. And, although blist_fill() was intended to support
fill operations with large counts, the old striping scheme never performed a
fill larger than the stripe size. Consequently, the misplacement of a
sanity check in blst_meta_fill() went undetected. Now, moving forward in
time to r118390, a new scheme for striping was introduced that maintained a
blist allocator per device, but as noted in r318995, swapoff_one() was not
fully and correctly converted to the new scheme. This change completes what
was started in r318995 by fixing the underlying bug in blst_meta_fill() that
stops swapoff_one() from simply performing a single blist_fill() operation.
MFC r319627
Starting in r118390, swaponsomething() began to reserve the blocks at the
beginning of a swap area for a disk label. However, neither r118390 nor
r118544, which increased the reservation from one to two blocks, correctly
accounted for these blocks when updating the variable "swap_pager_avail".
This change corrects that error.
MFC r319655
Originally, this file could be compiled as a user-space application for
testing purposes. However, over the years, various changes to the kernel
have broken this feature. This revision applies some fixes to get user-
space compilation working again. There are no changes in this revision
to code that is used by the kernel.
dexuan [Wed, 14 Jun 2017 13:23:40 +0000 (13:23 +0000)]
MFC: 319690
r319690
hyperv/pcib: use the device serial number as PCI domain
Currently the PCI domain is initialized with the instance GUID in
vmbus_pcib_attach(). It turns out the GUID can change across VM reboot,
while some users want a persistent value for PCI domain. The solution is
that we can change to use the device serial number, which starts with 1
and is unique within a VM.
Obtained from: Haiyang Zhang
Sponsored by: Microsoft
gjb [Thu, 8 Jun 2017 16:04:13 +0000 (16:04 +0000)]
MFC r313021, r319621:
r313021 (brd):
Force a few variables to be uppercase when they are evaluated.
r319621:
Ensure ${_CW} is uppercase when passing '-c' to mk-vmimage.sh,
otherwise with 'CLOUDWARE=vagrant-virtualbox', the path to the
configuration file may be incorrect.
dim [Tue, 6 Jun 2017 18:40:14 +0000 (18:40 +0000)]
Work around a gcc 4.2 bug on powerpc architectures, where using
-fdata-sections can cause bogus "section type conflict" errors. See
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=33168 for more information.
Since the upstream fixes are under GPLv3 and non-trivial, just disable
using function and data sections, at the cost of a slightly larger
executable.
Direct commit to stable/10, since the version of clang in head and
stable/11 cannot be built by gcc 4.2 at all.
gjb [Tue, 6 Jun 2017 14:46:45 +0000 (14:46 +0000)]
MFC r318943 (avg):
MFV r318942: 8166 zpool scrub thinks it repaired offline device
https://www.illumos.org/issues/8166
If we do a scrub while a leaf device is offline (via "zpool offline"),
we will inadvertently clear the DTL (dirty time log) of the offline
device, even though it is still damaged. When the device comes back
online, we will incompletely resilver it, thinking that the scrub
repaired blocks written before the scrub was started. The incomplete
resilver can lead to data loss if there is a subsequent failure of a
different leaf device.
The fix is to never clear the DTL of offline devices. Note that if a
device is onlined while a scrub is in progress, the scrub will be
restarted.
The problem can be worked around by running "zpool scrub" after
"zpool online".
See also https://github.com/zfsonlinux/zfs/issues/5806
hselasky [Sun, 4 Jun 2017 08:47:09 +0000 (08:47 +0000)]
MFC r319413:
Free hardware queue resource after port is stopped in the mlx4en(4)
driver. Else if the port is up the resource might still be busy and
the MTT free will fail.
hselasky [Sun, 4 Jun 2017 08:29:17 +0000 (08:29 +0000)]
MFC r319414:
Allow communication between functions on the same host when using the
mlx4en(4) driver in SRIOV mode.
Place a copy of the destination MAC address in the send WQE only under
SRIOV/eSwitch configuration or when the device is in selftest. This
allows communication between functions on the same host.
ngie [Fri, 2 Jun 2017 00:29:23 +0000 (00:29 +0000)]
MFC r318211:
Fix up previous commit
- Apply the logic to the FreeBSD block
- Fix a typo with the getconf(1) call that I would have caught, were
it not for the fact that I got the blocks wrong.
- Consolidate the hardcoded buffer sizes to the NetBSD block.
This would have been discovered had I run the test on a system where
PATH_MAX != 1024 (I don't have that at my disposal right at this moment).
ngie [Fri, 2 Jun 2017 00:27:08 +0000 (00:27 +0000)]
MFC r318210:
ssp_test:read:: query the value of MAXPATHLEN via getconf(1)
In the event the value of PATH_MAX was changed, the assumption that
MAXPATHLEN is 1024 (and hence the buffer length required to trigger
SSP to fail for read(2)) would be invalidated. Query getconf(1) for
the actual value of MAXPATHLEN via _XOPEN_PATH_MAX instead, and
increment the value by 1 to ensure that the SSP support tests the
stack smashing support properly.
slm [Thu, 1 Jun 2017 16:54:10 +0000 (16:54 +0000)]
MFC r318895: Fix several problems with mapping code in mps(4).
MFC r318896: Fix several problems with mapping code in mpr(4).
-Add several comments describing what the mapping code is doing.
-Added a callout timer to improve check for missing devices when discovery has
completed so that missing counts are incremented correctly.
-Fix problems with missing counts not being saved to the HBA.
-Update man pages mps(4) and mpr(4) to include a description of the use
use_phy_num sysctl variable.
-Remove channel field in the mapping structure because it's not used.
-Improve logging by using mps_dprint or mpr_dprint instead of printf and adding
more logging where appropriate.
-Add check for a bad index before writing mapping entries to controller.
-The high missing count check in the mapping table was using the incorrect
initial value, which could lead to a bad result.
-The usage of the IN_USE flag for volume mapping was changed to be more
intuitive, and was not being used correctly.
-The check for a free DPM entry was changed, as this was completely wrong.
-Updates to the missing count for volumes were not being done correctly, so this
function was completely rewritten.
-_mapping_add_to_removal_table() was overly complicated and incorrectly used, so
this function was rewritten.
-Missing counts for all devices were not being incremented properly, so this
functionality was added.
-The search for space in the mapping table for missing enclosures was not
calculating the found space correctly due to not breaking out of a loop when
required, and the num_found variable was not being reset when needed.
-Retries when a device fails to get added due to a full mapping table were
removed because this is unneccessary.
-mps_mapping_is_reinit_required() and mpr_mapping_is_reinit_required() were
removed because they were not being used.
-Some functions were renamed to avoid confusion between Target IDs and SAS IDs.
-_mapping_check_update_ir_mt_idx() was removed because it was overly
complicating volume mapping.
-The setting of the maxtargets variable was changed to include max volumes.
-The setting of the initiator_id variable was changed to be the invalid target
ID after all targets, including volumes. Previously, this was set to the last
valid target ID.
-Don't exclude target IDs of RAID components or check for a reuse of a target ID
for RAID components.
-Some endienness was added.
r295286:
Add sysctls for dumping out the device mapping tables. I'm finding this
useful for debugging device-target translation bugs.
MFC after: 3 days
Sponsored by: Netflix
r299367:
Remove NULL checks after M_WAITOK allocations from mps(4).
Reviewed by: asomers@
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D6296
r299369:
Remove NULL checks after M_WAITOK allocations from mpr(4).
Reviewed by: asomers@
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D6297
r308217:
Add a fallback to the device mapper logic. We've seen systems in the field
that are apparently misconfigured by the manufacturer and cause the mapping
logic to fail. The fallback allows drive numbers to be assigned based on the
PHY number that they're attached to. Add sysctls and tunables to overrid
this new behavior, but they should be considered only necessary for debugging.
r308301:
Record the LogInfo field when reporting the IOCStatus. Helps in
debugging errors.
Submitted by: slm
Obtained from: Netflix
MFC after: 3 days
r311958:
Print out the number of queues/MSIx vectors.
Sponsored by: Netflix
r312437:
Rework the debug print API. Event printing no longer gets special handling.
All of the printing from the tables file now has wrappers so that the
handling is cleaner and it's possible to print something out (say, during
development) without having to fight the global debug flags. This re-org
will also make it easier to have the tables be compiled out at build time
if desired.
Other than fixing some minor bugs, there are no user-visible changes from
this change
Sponsored by: Netflix, Inc.
Differential Revision: D9238
r318188:
Improve error messages during command timeout for the mpr and mps
drivers.
Sponsored by: Netflix
r318427:
Add tri-mode support (SAS/SATA/PCIe).
This includes NVMe device support and adds support for the following adapters:
SAS 3408
SAS 3416
SAS 3508
SAS 3516
SAS 3616
SAS 3708
SAS 3716
ngie [Thu, 1 Jun 2017 07:08:26 +0000 (07:08 +0000)]
MFC r314579,r314785:
r314579 (by np):
Add cxgbetool(8) to the base system.
Move cxgbetool from tools/tools to usr.sbin. Compile and install it on
platforms where cxgbe(4) is built by default. Knobs (WITH_CXGBETOOL and
WITHOUT_CXGBETOOL) have been added so that the user can override the
default setting.
ngie [Thu, 1 Jun 2017 06:41:27 +0000 (06:41 +0000)]
MFC r318436:
usr.bin/getconf: add some initial tests
Items tested via this commit are:
- Some basic POSIX constants.
- Some valid programming environments with -v.
- Some invalid programming environments via -v.
NOTE: this test makes assumptions about ILP32/LP32 vs LP64 that are
currently not true on all architectures to avoid hardcoding some
architectures in the tests. I'm working on improving getconf(1) to be
more sane about handling ILP32/LP32 vs LP64. Future commits are coming
soon to address this.
projects / FreeBSD / stable / 10.git / log