jeff [Tue, 19 Nov 2019 23:19:43 +0000 (23:19 +0000)]
Simplify anonymous memory handling with an OBJ_ANON flag. This eliminates
reudundant complicated checks and additional locking required only for
anonymous memory. Introduce vm_object_allocate_anon() to create these
objects. DEFAULT and SWAP objects now have the correct settings for
non-anonymous consumers and so individual consumers need not modify the
default flags to create super-pages and avoid ONEMAPPING/NOSPLIT.
kevans [Tue, 19 Nov 2019 23:12:43 +0000 (23:12 +0000)]
bcm2835_sdhci: various refactoring of DMA path
This round of refactoring is mostly about streamlining the interrupt handler
to make it easier to verify and reason about operations taking place while
trying to bring FreeBSD up on the RPi4.
vmaffione [Tue, 19 Nov 2019 21:10:44 +0000 (21:10 +0000)]
bhyve: virtio-net: disable receive until features are negotiated
This patch fixes a race condition where the receive callback is called
while the device is being reset. Since the rx_merge variable may change
during reset, the receive callback may operate inconsistently with what
the guest expects.
Also, get rid of the unused rx_vhdrlen variable.
bz [Tue, 19 Nov 2019 21:08:18 +0000 (21:08 +0000)]
nd6: sysctl
Move the SYSCTL_DECL to the top of the file. Move the sysctl function
before SYSCTL_PROC so that we don't need an extra function declaration in
the middle of the file.
bz [Tue, 19 Nov 2019 20:34:33 +0000 (20:34 +0000)]
nd6_rtr: re-sort functions
Resort functions within file in a way that they depend on each other as
that makes it easier to rework various things.
Also allows us to remove file local function declarations.
alc [Tue, 19 Nov 2019 19:05:05 +0000 (19:05 +0000)]
Achieve two goals at once: (1) Avoid an unnecessary broadcast TLB
invalidation in reclaim_pv_chunk(). (2) Prevent an "invalid ASID" assertion
failure in reclaim_pv_chunk(). The detailed explanation for this change is
provided by r354792.
bz [Tue, 19 Nov 2019 15:38:55 +0000 (15:38 +0000)]
Reduce the vnet_set module size of ip_mroute to allow loading as a module.
With VIMAGE kernels modules get special treatment as they need
to also keep the original values and make copies for each instance.
For that a few pages of vnet modspace are provided and the
kernel-linker and the VNET framework know how to deal with things.
When the modspace is (almost) full, other modules which would
overflow the modspace cannot be loaded and kldload will fail.
ip_mroute uses a lot of variable space, mostly be four big arrays:
set_vnet 0000000000000510 vnet_entry_multicast_register_if
set_vnet 0000000000000700 vnet_entry_viftable
set_vnet 0000000000002000 vnet_entry_bw_meter_timers
set_vnet 0000000000002800 vnet_entry_bw_upcalls
Dynamically malloc the three big ones for each instance we need
and free them again on vnet teardown (the 4th is an ifnet).
That way they only need module space for a single pointer and
allow a lot more modules using virtualized variables to be loaded
on a VNET kernel.
bz [Tue, 19 Nov 2019 14:53:13 +0000 (14:53 +0000)]
mld: fix epoch assertion
in6ifa_ifpforlinklocal() asserts the net epoch. The test case from r354832
revealed code paths where we call into the function without having
acquired the net epoch first and consequently we hit the assert.
This happens in certain MLD states during VNET shutdown and most people
normaly not notice this.
For correctness acquire the net epoch around calls to
mld_v1_transmit_report() in all cases to avoid the assertion firing.
dab [Tue, 19 Nov 2019 14:46:28 +0000 (14:46 +0000)]
Don't sanitize linker_set
The assumptions of linker_set don't play nicely with
AddressSanitizer. AddressSanitizer adds a 'redzone' of zeros around
globals (including those in named sections), whereas linker_set
assumes they are all packed consecutively like a pointer array. So:
let's annotate linker_set so that AddressSanitizer ignores it.
dougm [Tue, 19 Nov 2019 08:06:31 +0000 (08:06 +0000)]
Drop the extra argument from swp_pager_meta_ctl and have it do lookup
only. Rename it swp_pager_meta_lookup. Stop checking for obj->type
== swap there and assert it instead. Make the caller responsible for
the obj->type check.
Move the meta_ctl 'pop' functionality to swap_pager_unswapped, the
only place that uses it, and assume obj->type == swap there too.
cem [Tue, 19 Nov 2019 04:30:23 +0000 (04:30 +0000)]
unifdef(1): Improve worst-case bound on symbol resolution
Use RB_TREE to make some algorithms O(lg N) and O(N lg N) instead of O(N)
and O(N^2). Because N is typically small and the former linear array also has
great constant factors (as a property of CPU caching), this doesn't provide
material benefit most or all of the time.
While here, remove arbitrarily limit on number of macros understood.
This allows easy and care-free scaling of NUM_DMA_SEGS with proper-ish
calculations to make sure we can actually handle the number of segments we'd
like to handle on average so that performance comparisons can be easily made
at different values if/once we can actually handle it. It also makes it
helps the untrained reader understand more quickly the reasoning behind the
choice of maxsize/maxsegs/maxsegsize.
jhibbits [Tue, 19 Nov 2019 02:00:13 +0000 (02:00 +0000)]
powerpc/pmap: Remove an unused error from moea64_pvo_enter()
ENOENT is leftover from mmu_oea.c's moea_pvo_enter(), where it's used to
syncicache() on the first new mapping of a page. This sync is done
differently in OEA64.
jhibbits [Tue, 19 Nov 2019 01:28:06 +0000 (01:28 +0000)]
powerpc/booke pmap: Use the right 'tlbilx' form to invalidate TIDs
'tlbilxpid' is 'tlbilx 1, 0', while the existing form is 'tlbilx 0, 0',
which translates to 'tlbilxlpid', invalidating a LDPID. This effectively
invalidates the entire TLB, causing unnecessary reloads.
kevans [Mon, 18 Nov 2019 23:31:12 +0000 (23:31 +0000)]
sysent: regenerate after r354835
The lua-based makesyscalls produces slightly different output than its
makesyscalls.sh predecessor, all whitespace differences more closely
matching the source syscalls.master.
kevans [Mon, 18 Nov 2019 23:28:23 +0000 (23:28 +0000)]
Convert in-tree sysent targets to use new makesyscalls.lua
flua is bootstrapped as part of the build for those on older
versions/revisions that don't yet have flua installed. Once upgraded past
r354833, "make sysent" will again naturally work as expected.
kevans [Mon, 18 Nov 2019 23:21:13 +0000 (23:21 +0000)]
Add flua to the base system, install to /usr/libexec
FreeBSDlua ("flua") is a FreeBSD-private lua, flavored with whatever
extensions we need for base system operations. We currently support a subset
of lfs and lposix that are used in the rewrite of makesyscall.sh into lua,
added in r354786.
flua is intentionally written such that one can install standard lua and
some set of lua modules from ports and achieve the same effect.
linit_flua is a copy of linit.c from contrib/lua with lfs and lposix added
in. This is similar to what we do in stand/. linit.c has been renamed to
make it clear that this has flua-specific bits.
luaconf has been slightly obfuscated to make extensions more difficult. Part
of the problem is that flua is already hard enough to use as a bootstrap
tool because it's not in PATH- attempting to do extension loading would
require a special bootstrap version of flua with paths changed to protect
the innocent.
src.lua.mk has been added to make it easy for in-tree stuff to find flua,
whether it's bootstrap-flua or relying on PATH frobbing by Makefile.inc1.
bz [Mon, 18 Nov 2019 21:59:47 +0000 (21:59 +0000)]
icmpv6: Fix mbuf change in mld
After r354748 mld_input() can change the mbuf. The new pointer
is never returned to icmp6_input() and when passed to
icmp6_rip6_input() the mbuf may no longer valid leading to
a panic.
Pass a pointer to the mbuf to mld_input() so we can return an
updated version in the non-error case.
Add a test sending an MLD packet case which will trigger this bug.
kib [Mon, 18 Nov 2019 20:56:59 +0000 (20:56 +0000)]
bus_dma_dmar_set_buswide(9): KPI to indicate that the whole dmar
context should share page tables.
Practically it means that dma requests from any device on the bus are
translated according to the entries loaded for the bus:0:0 device.
KPI requires that the slot and function of the device be 0:0, and that
no tags for other devices on the bus were used.
The intended use are NTBs which pass TLPs from the downstream to the
host with slot:func of the downstream originator.
Reviewed and tested by: mav
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D22434
markj [Mon, 18 Nov 2019 20:03:28 +0000 (20:03 +0000)]
Set MALLOC_DEBUG_MAXZONES=1 in GENERIC-NODEBUG configurations.
The purpose of this option is to make it easier to track down memory
corruption bugs by reducing the number of malloc(9) types that might
have recently been associated with a given chunk of memory. However, it
increases fragmentation and is disabled in release kernels.
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
kevans [Mon, 18 Nov 2019 19:28:09 +0000 (19:28 +0000)]
bcm2835_sdhci: use a macro for interrupts we handle
This is just further simplification, very little functional change. In the
DMA interrupt handler, we *do* now acknowledge both DATA_AVAIL | SPACE_AVAIL
every time -- these operations are mutually exclusive, so while this is a
functional change, it's effectively a nop. Removing the 'mask' local allows
us to further simplify in a future change.
kevans [Mon, 18 Nov 2019 18:40:35 +0000 (18:40 +0000)]
bcm2835_sdhci: push DATA_END handling out of DMA interrupt path
This simplifies the DMA interrupt handler quite a bit. The sdhci framework
will call platform_finish_transfer() if it's received SDHCI_INT_DATA_END, so
we can take care of any final cleanup there and simply not worry about the
possibility of it ending in the DMA interrupt path.
markj [Mon, 18 Nov 2019 18:34:23 +0000 (18:34 +0000)]
Fix inconsistencies in anonymous DOF files.
The DOF file output by dtrace -A contains only the loadable sections.
However, as it was created by a call to dtrace_dof_create() without
flags, the original DOF was created with the loadable sections. The
result is that the DOF includes the section headers for the unloadable
sections (COMMENTS and UTSNAME) without these sections actually being
present. This is inconsistent.
A simple change to anon_prog() ensures that the missing sections are
present in the outputted DOF. Alternatively, the call to
dtrace_dof_create() could pass the DTRACE_D_STRIP flag stripping out the
loadable sections. As the unloadable sections contain info useful for
debugging purposes they haven't been stripped.
markj [Mon, 18 Nov 2019 18:25:51 +0000 (18:25 +0000)]
Group per-domain reservation data in the same structure.
We currently have the per-domain partially populated reservation queues
and the per-domain queue locks. Define a new per-domain padded
structure to contain both of them. This puts the queue fields and lock
in the same cache line and avoids the false sharing within the old queue
array.
Also fix field packing in the reservation structure. In many places we
assume that a domain index fits in 8 bits, so we can do the same there
as well. This reduces the size of the structure by 8 bytes.
Update some comments while here. No functional change intended.
Reviewed by: dougm, kib
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D22391
markj [Mon, 18 Nov 2019 18:22:41 +0000 (18:22 +0000)]
Widen the vm_page aflags field to 16 bits.
We are now out of aflags bits, whereas the "flags" field only makes use
of five of its sixteen bits, so narrow "flags" to eight bits. I have no
intention of adding a new aflag in the near future, but would like to
combine the aflags, queue and act_count fields into a single atomically
updated word. This will allow vm_page_pqstate_cmpset() to become much
simpler and is a step towards eliminating the use of the page lock array
in updating per-page queue state.
The change modifies the layout of struct vm_page, so bump
__FreeBSD_version.
dab [Mon, 18 Nov 2019 13:31:16 +0000 (13:31 +0000)]
Jail and capability mode for shm_rename; add audit support for shm_rename
Co-mingling two things here:
* Addressing some feedback from Konstantin and Kyle re: jail,
capability mode, and a few other things
* Adding audit support as promised.
The audit support change includes a partial refresh of OpenBSM from
upstream, where the change to add shm_rename has already been
accepted. Matthew doesn't plan to work on refreshing anything else to
support audit for those new event types.
avg [Mon, 18 Nov 2019 10:34:27 +0000 (10:34 +0000)]
fix up r354804, link zstreamdump with libzfs
Since r354804 libzpool depends on libzfs for get_system_hostid symbol.
Except for zstreamdump, all binaries linked with libzpool were already
linked with libzfs. So, zstreamdump is the only fall-out.
It's interesting that on amd64 not only I was able to successfully build
zstreamdump, I am able to run it despite having the unresolved symbol in
libzpool.
10701 Correct lock ASSERTs in vdev_label_read/write
illumos/illumos-gate@58447f688d5e308373ab16a3b129bc0ba0fbc154
https://github.com/illumos/illumos-gate/commit/58447f688d5e308373ab16a3b129bc0ba0fbc154
https://www.illumos.org/issues/10701
Port of ZoL commit: 0091d66f4e Correct lock ASSERTs in vdev_label_read/write
At a minimum, this fixes a blown assert during an MMP test run when running on
a DEBUG build.
11770 additional mmp fixes
illumos/illumos-gate@4348eb901228d2f8fa50bb132a34248e8662074e
https://github.com/illumos/illumos-gate/commit/4348eb901228d2f8fa50bb132a34248e8662074e
https://www.illumos.org/issues/11770
Port a few additional MMP fixes from ZoL that came in after our
initial MMP port. 4ca457b065 ZTS: Fix mmp_interval failure ca95f70dff zpool import progress kstat
(only minimal changes from above can be pulled in right now) 060f0226e6 MMP interval and fail_intervals in uberblock
Note from the committer (me).
I do not have any use for this feature and I have not tested it. I only
did smoke testing with multihost=off.
Please be aware.
I merged the code only to make future merges easier.
Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Portions contributed by: Tim Chase <tim@chase2k.com>
Portions contributed by: sanjeevbagewadi <sanjeev.bagewadi@gmail.com>
Portions contributed by: John L. Hammond <john.hammond@intel.com>
Portions contributed by: Giuseppe Di Natale <dinatale2@llnl.gov>
Portions contributed by: Prakash Surya <surya1@llnl.gov>
Portions contributed by: Brian Behlendorf <behlendorf1@llnl.gov>
Author: Olaf Faaland <faaland1@llnl.gov>
jhibbits [Sun, 17 Nov 2019 20:49:24 +0000 (20:49 +0000)]
powerpc: Re-add -Wno-redundant-decls to DPAA build flags
Since the DPAA code is from a third party, with minimal edits, there is no
intent to fix these specific warnings at this time. Hide these warnings to
prevent the noise from hiding real warnings.
alc [Sun, 17 Nov 2019 17:38:53 +0000 (17:38 +0000)]
Achieve two goals at once: (1) Avoid an unnecessary broadcast TLB
invalidation in pmap_remove_all(). (2) Prevent an "invalid ASID" assertion
failure in pmap_remove_all().
The architecture definition specifies that the TLB will not cache mappings
that don't have the "AF" bit set, so pmap_remove_all() needn't issue a TLB
invalidation for mappings that don't have the "AF" bit set.
We allocate ASIDs lazily. Specifically, we don't allocate an ASID for a
pmap until we are activating it. Now, consider what happens on a fork().
Before we activate the child's pmap, we use pmap_copy() to copy mappings
from the parent's pmap to the child's. These new mappings have their "AF"
bits cleared. Suppose that the page daemon decides to reclaim a page that
underlies one of these new mappings. Previously, the pmap_invalidate_page()
performed by pmap_remove_all() on a mapping in the child's pmap would fail
an assertion because that pmap hasn't yet been assigned an ASID. However,
we don't need to issue a TLB invalidation for such mappings because they
can't possibly be cached in the TLB.
kevans [Sun, 17 Nov 2019 14:08:19 +0000 (14:08 +0000)]
Add makesyscalls.lua, a rewrite of makesyscalls.sh
This currently requires a suitable lua + luafilesystem + luaposix from the
ports tree to build. Discussion is underway in D21893 to add a suitable lua
to the base system, cleverly disguised and out of the way of normal
consumers.
makesyscalls.sh is a good target for rewrite into lua as it's currently a
sh+sed+awk script that can be difficult to add on to, at times. For
instance, adding a new COMPAT* option (that mimicks the behaivor of most
other COMPAT* options) requires a fairly substantial amount of copy/paste;
see r352693 for instance. Attempts to generate part of the awk script for
COMPAT* handling was (very kindly) rejected with a desire to just rewrite
the script in a single language that can handle all of it.
jhibbits [Sat, 16 Nov 2019 16:36:20 +0000 (16:36 +0000)]
powerpcspe: Don't leak kernel registers in SPE dumps
save_vec_int() for SPE saves off only the high word of the register, leaving
the low word as "garbage", but really containing whatever was in the kernel
register at the time. This leaks into core dumps, and in a near future
commit also into ptrace. Instead, save the GPR in the low word in
save_vec_nodrop(), which is used only for core dumps and ptrace.
jhibbits [Sat, 16 Nov 2019 16:27:31 +0000 (16:27 +0000)]
powerpcspe: Mark asm statement in spe_save_reg_high as clobbering memory
Modern gcc errors that "'vec[0]' is used uninitialized in this function"
without us telling it that vec is clobbered. Neither clang nor gcc 4.2.1
error on the existing construct.
tuexen [Sat, 16 Nov 2019 11:57:12 +0000 (11:57 +0000)]
Improve TCP CUBIC specific after idle reaction.
The adjustments are inspired by the Linux stack, which has had a
functionally equivalent implementation for more than a decade now.
Submitted by: Richard Scheffenegger
Reviewed by: Cheng Cui
Differential Revision: https://reviews.freebsd.org/D18982
tuexen [Sat, 16 Nov 2019 11:37:26 +0000 (11:37 +0000)]
Implement a tCP CUBIC-specific after idle reaction.
This patch addresses a very common case of frequent application stalls,
where TCP runs idle and looses the state of the network.
Submitted by: Richard Scheffenegger
Reviewed by: Cheng Cui
Differential Revision: https://reviews.freebsd.org/D18954
scottl [Sat, 16 Nov 2019 00:26:42 +0000 (00:26 +0000)]
TSX Asynchronous Abort mitigation for Intel CVE-2019-11135.
This CVE has already been announced in FreeBSD SA-19:26.mcu.
Mitigation for TAA involves either turning off TSX or turning on the
VERW mitigation used for MDS. Some CPUs will also be self-mitigating
for TAA and require no software workaround.
Control knobs are:
machdep.mitigations.taa.enable:
0 - no software mitigation is enabled
1 - attempt to disable TSX
2 - use the VERW mitigation
3 - automatically select the mitigation based on processor
features.
machdep.mitigations.taa.state:
inactive - no mitigation is active/enabled
TSX disable - TSX is disabled in the bare metal CPU as well as
- any virtualized CPUs
VERW - VERW instruction clears CPU buffers
not vulnerable - The CPU has identified itself as not being
vulnerable
Nothing in the base FreeBSD system uses TSX. However, the instructions
are straight-forward to add to custom applications and require no kernel
support, so the mitigation is provided for users with untrusted
applications and tenants.
bz [Sat, 16 Nov 2019 00:17:35 +0000 (00:17 +0000)]
nd6: retire defrouter_select(), use _fib() variant.
Burn bridges and replace the last two calls of defrouter_select() with
defrouter_select_fib(). That allows us to retire defrouter_select()
and make it more clear in the calling code that it applies to all FIBs.
bz [Sat, 16 Nov 2019 00:02:36 +0000 (00:02 +0000)]
nd6_rtr:
Pull in the TAILQ_HEAD() as it is not needed outside nd6_rtr.c.
Rename the TAILQ_HEAD() struct and the nd_defrouter variable from
"nd_" to "nd6_" as they are not part of the RFC 3542 API which uses "ND_".
Ideally I'd like to also rename the struct nd_defrouter {} to "nd6_*"
but given that is used externally there is more work to do.
scottl [Fri, 15 Nov 2019 23:27:17 +0000 (23:27 +0000)]
Create a new sysctl subtree, machdep.mitigations. Its purpose is to organize
knobs and indicators for code that mitigates functional and security issues
in the architecture/platform. Controls for regular operational policy should
still go into places security, hw, kern, etc.
The machdep root node is inherently architecture dependent, but mitigations
tend to be architecture dependent as well. Some cases like Spectre do cross
architectural boundaries, but the mitigation code for them tends to be
architecture dependent anyways, and multiple architectures won't be active
in the same image of the kernel.
Many mitigation knobs already exist in the system, and they will be moved
with compat naming in the future. Going forward, mitigations should collect
in machdep.mitigations.
bz [Fri, 15 Nov 2019 23:12:19 +0000 (23:12 +0000)]
if_llatbl: change htable_unlink_entry() to early exist if no work to do
Adjust the logic in htable_unlink_entry() to the one in
htable_link_entry() saving a block indent and making it more clear
in which case we do not do any work.
mav [Fri, 15 Nov 2019 23:01:09 +0000 (23:01 +0000)]
Initialize *comp_update with valid value.
I've noticed that sometimes with enabled DMAR initial write from device
to this address is somehow getting delayed, triggering assertion due to
zero default being invalid.
mav [Fri, 15 Nov 2019 22:47:59 +0000 (22:47 +0000)]
Cleanup address range checks in ioat(4).
- Deduce allowed address range for bus_dma(9) from the hardware version.
Different versions (CPU generations) have different documented limits.
- Remove difference between address ranges for src/dst and crc. At least
docs for few recent generations of CPUs do not mention anything like that,
while older are already limited with above limits.
- Remove address assertions from arguments. While I do not think the
addresses out of allowed ranges should realistically happen there due to
the platforms physical address limitations, there is now bus_dma(9) to
make sure of that, preferably via IOMMU.
- Since crc now has the same address range as src/dst, remove crc_dmamap,
reusing dst2_dmamap instead.
Discussed with: cem
MFC after: 2 weeks
Sponsored by: iXsystems, Inc.
bz [Fri, 15 Nov 2019 21:55:41 +0000 (21:55 +0000)]
Remove now unused IPv6 macros and update docs.
After r354748-354750 all uses of the IP6_EXTHDR_CHECK() and
IP6_EXTHDR_GET() macros are gone from the kernel. IP6_EXTHDR_GET0()
was unused. Remove the macros and update the documentation.
bz [Fri, 15 Nov 2019 21:51:43 +0000 (21:51 +0000)]
IP6_EXTHDR_CHECK(): remove the last instances
While r354748 removed almost all IP6_EXTHDR_CHECK() calls, these
are not part of the PULLDOWN_TESTS.
Equally convert these IP6_EXTHDR_CHECK()s here to m_pullup() and remove
the extra check and m_pullup() in tcp_input() under isipv6 given
tcp6_input() has done exactly that pullup already.
bz [Fri, 15 Nov 2019 21:44:17 +0000 (21:44 +0000)]
netinet*: replace IP6_EXTHDR_GET()
In a few places we have IP6_EXTHDR_GET() left in upper layer protocols.
The IP6_EXTHDR_GET() macro might perform an m_pulldown() in case the data
fragment is not contiguous.
Convert these last remaining instances into m_pullup()s instead.
In CARP, for example, we will a few lines later call m_pullup() anyway,
the IPsec code coming from OpenBSD would otherwise have done the m_pullup()
and are copying the data a bit later anyway, so pulling it in seems no
better or worse.
Note: this leaves very few m_pulldown() cases behind in the tree and we
might want to consider removing them as well to make mbuf management
easier again on a path to variable size mbufs, especially given
m_pulldown() still has an issue not re-checking M_WRITEABLE().
bz [Fri, 15 Nov 2019 21:40:40 +0000 (21:40 +0000)]
netinet6: Remove PULLDOWN_TESTs.
Remove the KAME introduced PULLDOWN_TESTs which did not even
have a compile-time option in sys/conf to turn them on for a
custom kernel build. They made the code a lot harder to read
or more complicated in a few cases.
Convert the IP6_EXTHDR_CHECK() calls into FreeBSD looking code.
Rather than throwing the packet away if it would not fit the
KAME mbuf expectations, convert the macros to m_pullup() calls.
Do not do any extra manual conditional checks upfront as to
whether the m_len would suffice (*), simply let m_pullup() do
its work (incl. an early check).
Remove extra m_pullup() calls where earlier in the function or
the only caller has already done the pullup.
bz [Fri, 15 Nov 2019 21:19:06 +0000 (21:19 +0000)]
Allow per-file lex and yacc options.
In order to allow software with multiple (different) options
for lex and yacc add extra per-file options to the calls.
This is especially useful when one .l file needs -Pprefix.