https://www.illumos.org/issues/8269
It seems that currently normalization of stddev aggregation is done
incorrectly.
We divide both the sum of values and the sum of their squares by the
normalization factor. But we should divide the sum of squares by the
normalization factor squared to scale the original values properly.
FreeBSD note: the actual change was committed in r316853, this commit
adds the test files and record merge information.
https://www.illumos.org/issues/8269
It seems that currently normalization of stddev aggregation is done
incorrectly.
We divide both the sum of values and the sum of their squares by the
normalization factor. But we should divide the sum of squares by the
normalization factor squared to scale the original values properly.
https://www.illumos.org/issues/8269
It seems that currently normalization of stddev aggregation is done
incorrectly.
We divide both the sum of values and the sum of their squares by the
normalization factor. But we should divide the sum of squares by the
normalization factor squared to scale the original values properly.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Toomas Soome <tsoome@me.com>
Reviewed by: Igor Kozhukhov <igor@dilos.org>
Reviewed by: Andrew Stormont <andyjstormont@gmail.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Yuri Pankov <yuri.pankov@nexenta.com>
https://www.illumos.org/issues/8168
If we manage to export the pool on which we are creating a dataset (filesystem
or zvol) between entering libzfs`zfs_create() and libzfs`zpool_open() call (for
which we never check the return value) we end up dereferencing a NULL pointer
in libzfs`zpool_close().
This was discovered on ZFS on Linux. The same issue can be reproduced on
Illumos running in parallel:
while :; do zpool import -d /tmp testpool ; zpool export testpool ; done
while :; do zfs create testpool/fs; zfs destroy testpool/fs ; done
Eventually this will result in several core dumps like this one:
[root@52-54-00-d3-7a-01 /cores]# mdb core.zfs.4244
Loading modules: [ libumem.so.1 libc.so.1 libtopo.so.1 libavl.so.1
libnvpair.so.1 ld.so.1 ]
> ::stack
libzfs.so.1`zpool_close+0x17(0, 0, 0, 8047450)
libzfs.so.1`zfs_create+0x1bb(8090548, 8047e6f, 1, 808cba8)
zfs_do_create+0x545(2, 8047d74, 80778a0, 801, 0, 3)
main+0x22c(8047d2c, fef5c6e8, 8047d64, 8055a17, 3, 8047d70)
_start+0x83(3, 8047e64, 8047e68, 8047e6f, 0, 8047e7b)
>
Fix and reproducer (systemtap): https://github.com/zfsonlinux/zfs/pull/6096
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: loli10K <ezomori.nozomu@gmail.com>
kib [Fri, 9 Jun 2017 12:06:22 +0000 (12:06 +0000)]
Remove msdosfs -o large support.
Its purpose was to translate the values for msdosfs inode numbers,
which is calculated from the msdosfs structures describing the file,
into the range representable by 32bit ino_t. The translation acted
for filesystems larger than 128Gb, it reserved the range 0xf0000000
(FILENO_FIRST_DYN) to UINT32_MAX and remembered some arbitrary
translation of ino >= FILENO_FIRST_DYN into this range. It consumed
memory that could be only freed by unmount, and the translation was
not stable across remounts.
With ino_t type extended to 64 bit, there is no such issue and values
can be returned without compaction to 32bit. That is, for the native
environments, the translation layer is not necessary and adds
significant undeserved code complexity. For compat ABIs which use
32bit ino_t, the vfs.ino64_trunc_error sysctl provides some measures
to soften the failure mode when inode numbers truncation is not safe.
Discussed with: bde
Sponsored by: The FreeBSD Foundation
kib [Fri, 9 Jun 2017 11:17:08 +0000 (11:17 +0000)]
Enhance vfs.ino64_trunc_error sysctl.
Provide a new mode "2" which returns a special overflow indicator in
the non-representable field instead of the silent truncation (mode
"0") or EOVERFLOW (mode "1").
In particular, the typical use of st_ino to detect hard links with
mode "2" reports false positives, which might be more suitable for
some uses.
Discussed with: bde
Sponsored by: The FreeBSD Foundation
glebius [Thu, 8 Jun 2017 21:33:19 +0000 (21:33 +0000)]
When we are in UMA_STARTUP use startup_alloc() for any zone, not for
internal zones only. This allows to create new zones at early stages
of boot, without need to mark them as internal to UMA, which isn't
always true.
glebius [Thu, 8 Jun 2017 21:30:34 +0000 (21:30 +0000)]
Listening sockets improvements.
o Separate fields of struct socket that belong to listening from
fields that belong to normal dataflow, and unionize them. This
shrinks the structure a bit.
- Take out selinfo's from the socket buffers into the socket. The
first reason is to support braindamaged scenario when a socket is
added to kevent(2) and then listen(2) is cast on it. The second
reason is that there is future plan to make socket buffers pluggable,
so that for a dataflow socket a socket buffer can be changed, and
in this case we also want to keep same selinfos through the lifetime
of a socket.
- Remove struct struct so_accf. Since now listening stuff no longer
affects struct socket size, just move its fields into listening part
of the union.
- Provide sol_upcall field and enforce that so_upcall_set() may be called
only on a dataflow socket, which has buffers, and for listening sockets
provide solisten_upcall_set().
o Remove ACCEPT_LOCK() global.
- Add a mutex to socket, to be used instead of socket buffer lock to lock
fields of struct socket that don't belong to a socket buffer.
- Allow to acquire two socket locks, but the first one must belong to a
listening socket.
- Make soref()/sorele() to use atomic(9). This allows in some situations
to do soref() without owning socket lock. There is place for improvement
here, it is possible to make sorele() also to lock optionally.
- Most protocols aren't touched by this change, except UNIX local sockets.
See below for more information.
o Reduce copy-and-paste in kernel modules that accept connections from
listening sockets: provide function solisten_dequeue(), and use it in
the following modules: ctl(4), iscsi(4), ng_btsocket(4), ng_ksocket(4),
infiniband, rpc.
o UNIX local sockets.
- Removal of ACCEPT_LOCK() global uncovered several races in the UNIX
local sockets. Most races exist around spawning a new socket, when we
are connecting to a local listening socket. To cover them, we need to
hold locks on both PCBs when spawning a third one. This means holding
them across sonewconn(). This creates a LOR between pcb locks and
unp_list_lock.
- To fix the new LOR, abandon the global unp_list_lock in favor of global
unp_link_lock. Indeed, separating these two locks didn't provide us any
extra parralelism in the UNIX sockets.
- Now call into uipc_attach() may happen with unp_link_lock hold if, we
are accepting, or without unp_link_lock in case if we are just creating
a socket.
- Another problem in UNIX sockets is that uipc_close() basicly did nothing
for a listening socket. The vnode remained opened for connections. This
is fixed by removing vnode in uipc_close(). Maybe the right way would be
to do it for all sockets (not only listening), simply move the vnode
teardown from uipc_detach() to uipc_close()?
jhb [Thu, 8 Jun 2017 21:06:18 +0000 (21:06 +0000)]
Add explicit handling for requests with an empty payload.
- For HMAC requests, construct a special input buffer to request an empty
hash result.
- For plain cipher requests and requests that chain an AES cipher with an
HMAC, fail with EINVAL if there is no cipher payload. If needed in
the future, chained requests that only contain AAD could be serviced as
HMAC-only requests.
- For GCM requests, the hardware does not support generating the tag for
an AAD-only request. Instead, complete these requests synchronously
in software on the assumption that such requests are rare.
jtl [Thu, 8 Jun 2017 20:47:18 +0000 (20:47 +0000)]
With EARLY_AP_STARTUP enabled, we are seeing crashes in softclock_call_cc()
during bootup. Debugging information shows that softclock_call_cc() is
trying to execute the vt_consdev.vd_timer callout, and the callout
structure contains a NULL c_func.
This appears to be due to a race between vt_upgrade() running
callout_reset() and vt_resume_flush_timer() calling callout_schedule().
Fix the race by ensuring that vd_timer_armed is always set before
attempting to (re)schedule the callout.
jtl [Thu, 8 Jun 2017 20:41:28 +0000 (20:41 +0000)]
Add the infrastructure to support loading multiple versions of TCP
stack modules.
It adds support for mangling symbols exported by a module by prepending
a string to them. (This avoids overlapping symbols in the kernel linker.)
It allows the use of a macro as the module name in the DECLARE_MACRO()
and MACRO_VERSION() macros.
It allows the code to register stack aliases (e.g. both a generic name
["default"] and version-specific name ["default_10_3p1"]).
With these changes, it is trivial to compile TCP stack modules with
the name defined in the Makefile and to load multiple versions of the
same stack simultaneously. This functionality can be used to enable
side-by-side testing of an old and new version of the same TCP stack.
It also could support upgrading the TCP stack without a reboot.
emaste [Thu, 8 Jun 2017 20:06:09 +0000 (20:06 +0000)]
arm64: add ".arch armv8-a+crc" to allow use of crc instructions
With Clang 5.0 the .arch directive is required, otherwise Clang
complains "error: instruction requires: crc".
This was reported in D10499 but not added initially, because clang 3.8
available on a ref machine reported unknown directive. Clang 4.0 allows
but does not require the directive.
Submitted by: andrew
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
asomers [Thu, 8 Jun 2017 19:09:55 +0000 (19:09 +0000)]
Add tests for ln(1)
* Verify that when creating a hard link to a symbolic link, '-L' option
creates a hard link to the target of the symbolic link
* Verify that when creating a hard link to a symbolic link, '-P' option
creates a hard link to the symbolic link itself
* Verify that if the target file already exists, '-f' option unlinks it so
that link may occur
* Verify that if the target file or directory is a symbolic link, '-shf'
option prevents following the link
* Verify that if the target file or directory is a symbolic link, '-snf'
option prevents following the link
* Verify that '-s' option creates a symbolic link
* Verify that '-w' option produces a warning if the source of a symbolic
link does not currently exist
zbb [Thu, 8 Jun 2017 16:55:58 +0000 (16:55 +0000)]
Restore DTS node of PCIe controller for A38X boards
Add pcie-controller node as a bus-parent of pcie nodes for Armada38x
boards. This reduces diff between Linux and FreeBSD PCIe device tree
representation to the minimum. This commit also allows for using multiple
PCIe ports, thanks to the recent driver updates, which support such
hierarchy. Restore original PCIe nodes in armada-385.dtsi and
apply necessary changes in hitherto unused armada-380.dtsi.
Submitted by: Michal Mazur <mkm@semihalf.com>
Marcin Wojtas <mw@semihalf.com>
Obtained from: Semihalf
Sponsored by: Stormshield, Netgate
Differential revision: https://reviews.freebsd.org/D10907
zbb [Thu, 8 Jun 2017 16:54:02 +0000 (16:54 +0000)]
Support multi-port PCIe hierarchy in Marvell boards DTS
This commit is another part of preparation for PCIe multi-port
support for Marvell SoCs. Some device trees include pcie-controller
node as a bus-parent of pcie nodes. This patch adds support for
new bus, collects and configures device informations and finally
adds PCIB devices as a childs of pcie-controller in Newbus hierarchy.
zbb [Thu, 8 Jun 2017 16:51:46 +0000 (16:51 +0000)]
Fix PCIe window decoding on Armada 38x
Original PCIe nodes for Marvell SoCs consists of ports' nodes
under main controller node. In order to properly parse
this kind of representation in DT a mechanism for traversing
through the tree required an update. Moreover, processing FDT
data consisting of more than 2 cells had to be fixed,
because the 'reg' property of mrvl,pcie node have additional
parameter in front of 64-bit address. It should be skipped
by default. This commit works properly with old mrvl,pcie
representation for Kirkwood and ArmadaXP SoCs.
Submitted by: Wojciech Macek <wma@semihalf.com>
Michal Mazur <mkm@semihalf.com>
Obtained from: Semihalf
Sponsored by: Stormshield, Netgate
Differential revision: https://reviews.freebsd.org/D10905
zbb [Thu, 8 Jun 2017 16:48:09 +0000 (16:48 +0000)]
Enable MBUS bridge configuration in mv_rtc driver
This patch fixes sporadic problems with updating time
with mv_rtc driver by configuring access to it via MBUS.
For this purpose already existing second set of resources
in rtc@3800 node of Armada 38x DT is used.
zbb [Thu, 8 Jun 2017 16:46:38 +0000 (16:46 +0000)]
Add reset capability to mv_rtc driver
This commit enables optional reset of the RTC, in case
its registers' contents did not sustain the reboot or power-off/on
sequence. Without it, further usage of RTC is impossible
(e.g. writing values to RTC_TIME register will not succeed).
The reset is performed only if Clock Correction register
does not comprise RTC_NOMINAL_TIMING, what helps to distinguish,
whether the software configured RTC before or it comprises
the default value.
jhb [Thu, 8 Jun 2017 16:18:41 +0000 (16:18 +0000)]
Fix an off-by-one error in the VM page array on some systems.
r31386 changed how the size of the VM page array was calculated to be
less wasteful. For most systems, the amount of memory is divided by
the overhead required by each page (a page of data plus a struct vm_page)
to determine the maximum number of available pages. However, if the
remainder for the first non-available page was at least a page of data
(so that the only memory missing was a struct vm_page), this last page
was left in phys_avail[] but was not allocated an entry in the VM page
array. Handle this case by explicitly excluding the page from
phys_avail[].
alc [Thu, 8 Jun 2017 15:48:54 +0000 (15:48 +0000)]
When allocating swap blocks, if the available number of free blocks in a
subtree is already zero, then setting the "largest contiguous free block"
hint for that subtree to anything other than zero makes no sense. To be
clear, assigning a value to the hint that is too large is not a correctness
problem, only a pessimization.
Dragonfly BSD has applied the same change to blst_meta_alloc() but not
blst_meta_fill().
phil [Thu, 8 Jun 2017 13:04:01 +0000 (13:04 +0000)]
Import libxo-0.8.0:
- addition of --libxo colors=xxxxx color map (so I never see "blue")
- fix bugs from -fsanitize=address and =undefined
- utf-8 changes (remove support fore 6 byte utf-8 values, which are "historical")
- add comments
- fix man pages
- update test cases
Submitted by: phil
Reviewed by: sjg
Approved by: sjg (mentor)
dexuan [Thu, 8 Jun 2017 12:11:30 +0000 (12:11 +0000)]
hyperv/pcib: use the device serial number as PCI domain
Currently the PCI domain is initialized with the instance GUID in
vmbus_pcib_attach(). It turns out the GUID can change across VM reboot,
while some users want a persistent value for PCI domain. The solution is
that we can change to use the device serial number, which starts with 1
and is unique within a VM.
Obtained from: Haiyang Zhang
MFC after: 1 day
Sponsored by: Microsoft
glebius [Thu, 8 Jun 2017 06:16:47 +0000 (06:16 +0000)]
Fix a degenerate case when soisdisconnected() would call soisconnected().
This happens when closing a socket with upcall, and trace is: soclose()->
... protocol ... -> soisdisconnected() -> socantrcvmore_locked() ->
sowakeup() -> soisconnected().
Right now this case is innocent for two reasons. First, soisconnected()
doesn't clear SS_ISDISCONNECTED flag. Second, the mutex to lock the
socket is the socket receive buffer mutex, and sodisconnected() first
disables the receive buffer. But in future code, the mutex to lock
socket is different to buffer mutex, and we would get undesired mutex
recursion.
The fix is to check SS_ISDISCONNECTED flag before calling upcall.
glebius [Thu, 8 Jun 2017 05:12:11 +0000 (05:12 +0000)]
Improve this unit test: make sure that the accept filter actually works.
Before this test just checked scenario of setting and removing the accept
filter at different states of the socket. Now it also checks that accept
filter works: we connect to the server, and then check that we can't accept,
then we send 1 byte of data and check again.
jhb [Thu, 8 Jun 2017 04:31:15 +0000 (04:31 +0000)]
Decode arguments passed to extended attribute related system calls.
The cmd argument passed to extattrctl() is not decoded as a string constant
but is just printed in hex. The value is filesystem-specific but in
practice is only used with UFS1 filesystems.
bapt [Thu, 8 Jun 2017 00:41:25 +0000 (00:41 +0000)]
Fix src.conf.5 generation
When makeman is generating src.conf(5) it tries to test all variation of options
including WITH_DIRDEPS_BUILD. it results in an error when filemon(4) is not
loaded.
Export variables that are needed to prevent this behaviour.
bapt [Wed, 7 Jun 2017 23:00:34 +0000 (23:00 +0000)]
Remove groff from base
All manpages in base are now compatible with mandoc(1), all roff documentation
will be relocated in the doc tree. man(1) can now use groff from the ports tree
if it needs.
Also remove checknr(1) and colcrt(1) which are only useful with groff.
ngie [Wed, 7 Jun 2017 21:03:27 +0000 (21:03 +0000)]
Add an MLINK for atf_check(1) -> atf-check(1)
This is being done to make the documentation for atf-check(1) easier to find/more
intuitive for new users, because atf_check is the atf-run(1) shell version of the
standalone atf-check(1) command, which is used in atf-sh(3) test programs.
jhibbits [Wed, 7 Jun 2017 18:08:11 +0000 (18:08 +0000)]
Add more #ifdef arch checks to the linuxkpi
arm, mips, and powerpc all implement pmap_mapdev_attr() and pmap_unmapdev(),
so add those archs to the checks. powerpc also includes the atomic_swap_*()
functions, so add that to the supported list as well. Not tested except by
compiling powerpc.
alc [Wed, 7 Jun 2017 16:04:34 +0000 (16:04 +0000)]
Originally, this file could be compiled as a user-space application for
testing purposes. However, over the years, various changes to the kernel
have broken this feature. This revision applies some fixes to get user-
space compilation working again. There are no changes in this revision
to code that is used by the kernel.
alc [Tue, 6 Jun 2017 16:52:07 +0000 (16:52 +0000)]
Starting in r118390, swaponsomething() began to reserve the blocks at the
beginning of a swap area for a disk label. However, neither r118390 nor
r118544, which increased the reservation from one to two blocks, correctly
accounted for these blocks when updating the variable "swap_pager_avail".
This change corrects that error.
ngie [Tue, 6 Jun 2017 16:04:27 +0000 (16:04 +0000)]
Add basic tests for echo(1)
Verify that echo(1) does not...
- ... print the trailing newline character with option '-n'.
- ... print the trailing newline character when '\c' is appended to
the end of the string.
gjb [Tue, 6 Jun 2017 14:08:54 +0000 (14:08 +0000)]
Ensure ${_CW} is uppercase when passing '-c' to mk-vmimage.sh,
otherwise with 'CLOUDWARE=vagrant-virtualbox', the path to the
configuration file may be incorrect.
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
alc [Tue, 6 Jun 2017 03:32:17 +0000 (03:32 +0000)]
When the function blist_fill() was added to the kernel in r107913, the swap
pager used a different scheme for striping the allocation of swap space
across multiple devices. And, although blist_fill() was intended to support
fill operations with large counts, the old striping scheme never performed a
fill larger than the stripe size. Consequently, the misplacement of a
sanity check in blst_meta_fill() went undetected. Now, moving forward in
time to r118390, a new scheme for striping was introduced that maintained a
blist allocator per device, but as noted in r318995, swapoff_one() was not
fully and correctly converted to the new scheme. This change completes what
was started in r318995 by fixing the underlying bug in blst_meta_fill() that
stops swapoff_one() from simply performing a single blist_fill() operation.
Reviewed by: kib
MFC after: 5 days
Differential Revision: https://reviews.freebsd.org/D11043
alc [Mon, 5 Jun 2017 17:14:16 +0000 (17:14 +0000)]
Halve the memory being internally allocated by the blist allocator. In
short, half of the memory that is allocated to implement the radix tree is
wasted because we did not change "u_daddr_t" to be a 64-bit unsigned int
when we changed "daddr_t" to be a 64-bit (signed) int. (See r96849 and
r96851.)
kib [Mon, 5 Jun 2017 11:40:30 +0000 (11:40 +0000)]
Add sysctl vfs.ino64_trunc_error controlling action on truncating
inode number or link count for the ABI compat binaries.
Right now, and by default after the change, too large 64bit values are
silently truncated to 32 bits. Enabling the knob causes the system to
return EOVERFLOW for stat(2) family of compat syscalls when some
values cannot be completely represented by the old structures. For
getdirentries(2), knob skips the dirents which would cause non-trivial
truncation of d_ino.
EOVERFLOW error is specified by the X/Open 1996 LFS document
('Adding Support for Arbitrary File Sizes to the Single UNIX
Specification').
Based on the discussion with: bde
Sponsored by: The FreeBSD Foundation
jhb [Mon, 5 Jun 2017 05:25:50 +0000 (05:25 +0000)]
Decode arguments to dup, dup2, getdirentries, pread, and pwrite.
- dup and dup2 print fd arguments in decimal.
- pread and pwrite are similar to read and write with the addition of the
file offset.
- getdirentries displays the output entries as a string for now and also
prints the value returned in *basep. Eventually the buffer for
getdirentries should perhaps be decoded as an array of dirent
structures.
PR: 214885
Submitted by: Jonathan de Boyne Pollard <J.deBoynePollard-newsgroups@NTLWorld.COM>
bdrewery [Mon, 5 Jun 2017 05:17:39 +0000 (05:17 +0000)]
META_MODE: Add framework to force rebuilding for major ABI changes.
Normally META_MODE ignores host files for "meta mode" decisions on whether a
file should be rebuilt or not. This is because a simple installworld can
update timestamps and cause the next build to rebuild all host tools, when the
previous ones may not have any changes in the source tree. These tools are
normally still ABI compatible. They are only rebuilt if NO_META_IGNORE_HOST is
set from the workaround/hack in r301467.
One of the major problems with this is when a host tool has objects spread
across many revisions that have mixed-ABI. For example, if struct stat were to
change on the host, some objects for a tool may have different ideas of that
struct's definition. If just 1 source file were modified and rebuilt and
linked into the tool, then that toll will have mixed-ABI objects and crash.
This exact thing happened with the ino64 commit in r301467 followed by a
trivial update to libbfd in r318750. The resulting binary would crash in
buildworld.