xff v0.8.0

Development history

Resolved design records and investigations moved out of the actionable backlog. For current work, see
TODO.md.

This preserves settled decisions, shipped implementation notes, and investigations whose context
would otherwise be lost when the actionable backlog is simplified.

The former combined roadmap and completion ledger through PR #683 is retained separately in
history-roadmap.md.

Block policy category naming

Use --block-policy-categories=LIST for the per-file selection of separate archive, temp, and
output controls. Parser, policy validation, registry help, reference documentation, and complete-file
fixtures share the spelling. The directive selects controls without granting or blocking operations;
category expansion and enforcement are unchanged. Unsupported spellings remain errors.

Required globals and optional named sections

The globals-requirement adjustment replaces the four require-config spellings with
--require-system-globals / --no-require-system-globals and
--require-user-globals / --no-require-user-globals. Skip flags always exclude named sections;
existing globals remain active by default and are excluded only when the applicable policy permits
it. No separate skip-globals flags or compatibility aliases are introduced. System policy for user
globals remains authoritative even when the system's ordinary globals are skipped.

The shared resolver applies the policy before section selection and safety expansion. Policy gating,
action arming, user admission decisions, and autoload globals permission therefore see retained
required globals. --no-config still disables autoloading. Files are read and validated even when
ordinary globals may be skipped. Complete-file tests verify deletion refusal under every skip form,
per-tier named-section exclusion, explicit-file selection, optional globals, system precedence, and
rejection of unsupported spellings. Missing automatic files remain normal.

Configuration validation and documentation audit

The follow-up to PR #820 found invalid global lines could discard a mandatory block while the
invocation continued, unknown selected names were ignored, bootstrap-only flags were accepted in
INI without effect, and INI pager settings did not reach listing output. Validation now fails
before actions for invalid globals and undefined explicit/composed selectors; inactive invalid named
sections remain disabled atomically. Registry metadata rejects bootstrap flags in INI and documents
config-only permission directives individually. Listing pager selection uses resolved configuration.
Trusted and explicit configuration readers reject special files before reading, matching autoload's
regular-file requirement while retaining intentional trusted/explicit symlinks to regular files.

Complete-file regressions cover each source tier, deletion refusal on malformed policy, selector
composition, pager precedence, and special-file rejection. The design overview now describes current
flag types, configuration, safety boundaries, and implementation contracts. Pager allowlisting remains
a separate design discussion; execution blocks deliberately do not govern pager commands.

The verification map for this audit is:

Local macOS all-extras tests, Linux GCC tests, and changed-file clang-tidy passed. Linux LLVM patch
coverage was 96.47% lines and 92.31% branches without weakening thresholds. Release preparation for
v0.4.0 passed in an isolated commit snapshot, including version stamping, module verification, and
release-note generation. Full CI on the final PR revision and staged release-artifact smoke tests
remain the integration gates; local results alone do not establish release readiness.

.xffrc admission order

The release quick check identified admission resolving selected user sections in declaration order.
Admission now consumes config-only controls through the shared ordered resolver, preserving CLI
selection, in-place composition, one-time application, and authoritative system denial. Regression
coverage includes both INI declaration orders, reversed selectors, literal command arguments,
explicit-file self-admission, and denial before autoload discovery probes a root. Runtime resolution
still omits config-only controls. This handles the confirmed admission inconsistency; the broader
configuration audit remains on the roadmap before release.

Opt-in .xffrc autoloading

Handled the first safety follow-up item: root-only --rc, recursive --rc+, and default-off
--rc-, with deterministic discovery before execution through the VFS. Added the config-only
rc-globals permission pair; unsectioned content without a grant is rejected rather than ignored.
Autoloaded files share INI parsing, composition, validation, and non-arming safety gating with
explicit files. Help and the configuration design describe scope, precedence,
permissions, examples, and discovery costs. Tests cover in-memory discovery and real CLI files.
Also moved the explain shell suite's runner below its previously unregistered directory-safety
test and corrected that test's regex assertion.

Directory-scoped safety controls

Completed the safety follow-up's temp/output scope work: extended per-INI policy translation,
config-only roots with system precedence, recursive descendant scope with protected roots,
separate directory creation/deletion controls, and descriptor-relative mutation routing.
Scoped overwrite replaces the directory entry to preserve other hard links. Archive publication
rechecks its destination; owned scratch cleanup uses retained handles. Real CLI tests cover INI
selection, scoped output, root protection, collision handling, archive publication, and dry-run.
The core and extra regression suite passed (184 tests) using Apple's linker locally because the
installed macOS SDK's new architecture tags are not recognized by hermetic LLD. The repository
linker configuration is unchanged. Remaining roadmap items are explicit rc autoloading and the
full configuration audit. Current behavior is documented in
Directory-scoped safety controls.

Resolved decisions

Sanitizer verification: what runs where

MSan CONFIRMED on x86_64 Linux, 2026-08-16: 124/124 Bazel test targets passed with no
MemorySanitizer findings. Nine mounting cases were skipped by design (6 in fuse_server_test,
1 mount-factory case in fuse_register_test, 2 CLI --archive-mount cases), for the reason in
"Why every mounting test skips under MSan" below. Everything that does NOT cross into the
uninstrumented system libfuse ran and passed: archive parsing, the FUSE loader and API logic,
mount lifecycle, registration, and the ownership tests.

So the honest statement of coverage is: no MSan defect exists in our own code, and real FUSE
kernel-mount behaviour is not exercised under MSan at all. That is not a hole left open - it is
the deliberate boundary of what MSan can say about a dlopened library it did not instrument. The
kernel path has its own unsanitized and TSan coverage (CI, plus tools/fuse_linux_test.sh).

Reproducing it

An x86_64 Linux machine with /dev/fuse (an ordinary VM is fine; sudo only for two
packages). aarch64 will NOT do - see "why" below. Then:

sudo apt-get install -y fuse3 libfuse3-3    # the runtime the loader dlopens + the unmount helper
git clone https://github.com/mboworks/xff && cd xff
# The MSan cell (this is the command that was run):
bazel test //... $(tools/extras.py --wildcards) --config=xff_docs \
  --config=clang --config=msan --config=xff_fuse_tests_required

# The TSan cell, same shape (already green in CI, cheap to confirm):
bazel test //... $(tools/extras.py --wildcards) --config=xff_docs \
  --config=clang --config=tsan --config=xff_fuse_tests_required

Expected: all tests pass, with every MOUNTING test reporting SKIPPED under --config=msan
(see below for why that is correct rather than a cop-out).

Why an x86_64 host specifically

MSan false-positives on anything it did not instrument, so the C++ standard library must be
instrumented too. --config=msan gets that from toolchains_llvm's msan feature, which swaps in
the instrumented libc++ overlay fetched by MSAN_LIBCXX_URL in bazelmod/llvm.MODULE.bazel - and
that overlay is published for x86_64-linux only. There is no aarch64 build to fetch.

Emulation was tried and does not work. tools/fuse_linux_test.sh msan forces a
--platform linux/amd64 container, which on Apple silicon means qemu; the build gets as far as
linking and then dies with clang: error: unable to execute command: No such file or directory

What IS verified locally, and how

tools/fuse_linux_test.sh (from a mac, via colima/docker) exists because macOS has no fuse3, so
every mounting test skips there and bazel test goes green without executing a line of the kernel
path. Two modes work on aarch64:

Note the container needs git installed (MODULE.bazel pulls toolchains_llvm and
hedron_compile_commands through git_repository, so a cold cache cannot even compute the repo
mapping) and roughly 25 GB of free VM disk for the hermetic LLVM.

Why every mounting test skips under MSan (and why that is not a dodge)

A mount runs through the dlopened system libfuse3, which MSan did not instrument, so every byte
libfuse writes reads back as uninitialized and the process dies inside fuse_opt_parse. There is
nothing to fix in our code: the report is about libfuse's memory, not ours. The skips are therefore
deliberate and keyed off --config=msan in one place per language:

To check the guards compile and fire without an MSan toolchain at all:
bazel test @xff_fuse//... --config=xff_full --copt=-DMEMORY_SANITIZER (the factory case must
report SKIPPED and the rest must pass), and
bazel test //xff/cli:full_extras_test --config=xff_full --//xff:xff_msan=true (both mount cases
must print the skip line, and must NOT print it without the flag).

Things that bit us, so nobody re-derives them

Result-set shaping: -first, -top, -collect, --max-results (design of record, 2026-08-17)

Ratified with the user. Supersedes the --top=N / --fuzzy-cutoff sketch: the first three are not
globals. They are expression primaries, because the interesting behaviour is per-invocation and
positional, which a whole-run flag cannot express. --max-results is the aggregate exception.

The insight that shapes everything else

A test that returns FALSE removes the entry from EVERYTHING downstream - the summary, the
histograms, the count, every action. So a truncating test can never also mean "summarise all of it
but show me a few": by the time the summary runs, the entries are gone. That is not a wrinkle to
work around; it is why -collect exists.

The vocabulary

Order is what selects the reading

-collect -top 10 -ls --summary   # collect all, list the 10 best, summary over ALL
-top 10 -collect --summary       # collect only the 10, summary over those, and NO listing

The second prints nothing extra because -collect is an action, so the implicit print is
suppressed and nothing else asked to print. Summary-only falls out of the existing rule instead of
needing a --quiet. This is the whole reason these are primaries: a position-independent global
(AGENTS.md hoists -- globals deliberately) makes both spellings identical and neither reading
expressible.

--max-results - and when it is pointless

The one genuine global here: an aggregate output ceiling. Call out plainly that it is only
irreducible when MULTIPLE capped filters are active - with a single -first 10 the cap already IS
ten, and with no capped filter it is just -first N spelled as a global. Its unique job:

xff . \( -type f -first 10 \) -o \( -type d -first 5 \) --max-results 12

Fifteen pass the filters; only an aggregate bound can say twelve. It caps OUTPUT and does not stop
the walk, because a summary or count that silently went partial is the same failure as the one above,
one level up; an early stop stays an explicit opt-in.

Build order (by machinery, not preference)

  1. -first N - immediate truth, no deferral, per-instance counter. Standalone. SHIPPED (#559).

  2. -collect[=NAME] - an action; makes --summary read the collection. SHIPPED. Two decisions
    the design left implicit, both settled by building it: --histogram switches its source together
    with --summary (a run where one reduced the collection and the other reduced the matches would
    report two different totals for one walk), and the collection OWNS its entries rather than storing
    the walk's Visit, whose path/name/root are borrowed views and whose metadata is a reference. The
    entry also keeps Visit::fs_owner, so collecting an archive member cannot outlive its reader -
    the same lifetime bug ThreadSanitizer caught in --archive-mount. --buffer now bounds the
    collection (a row window or a byte budget over the stored path/name/root text), and exceeding it
    is an ERROR (exit 2, naming the flag) rather than a silent truncation - a summary computed over
    part of the walk is indistinguishable from a correct one, which is the same reason
    --max-results caps output without stopping the walk. The DEFAULT is deliberately no cap: any
    number picked here would be a guess, and the point of the budget is that a run which would
    exhaust memory says so instead. A measured default is the open question if one is ever wanted.

    Name reuse: the ! modifier, not an override flag. A NAME is an identifier
    ([A-Za-z_][A-Za-z0-9_]*), which is what reserves punctuation for modifiers, and a node that
    reuses a name an earlier node took must say so with !:
    \( -type f -collect:all \) -o \( -type d -collect:!all \). This replaced the
    --collect-override / --capture-override globals outright, and the reason is the same one that
    made these primaries rather than globals: a whole-run flag loosens EVERY -capture / -collect in
    the command, including the ones the author never thought about, while the modifier loosens exactly
    the node it is written on. --capture-override is GONE (it was never released), and -capture
    now takes -capture:!NAME too. Settled with the user 2026-08-17.

  3. -top N SHIPPED (2026-08-20). Exact evaluator-level deferral, not a buffered print trick:
    each node gathers the scored entries that actually reach it, selects its stable top N after the
    walk, then replays the expression from memoized prefixes. Actions and stateful tests to the left
    therefore run once during traversal; everything to the right runs only after that node's decision.
    Several nodes resolve in expression order, so a candidate routed into a later -top only after
    losing an earlier one joins the later node BEFORE its selection - its candidate set is complete,
    not an early snapshot. Stored visits own their strings and archive filesystem, as collections do.
    A score must precede the node on every reachable path; model and threshold must be uniform (bare
    means 0%), and malformed / negative counts fail before traversal while zero keeps none. Complex
    end-to-end coverage pins exact ranking, before/after actions, independent nodes, late candidates,
    collection order, invalid domains, and zero.

  4. --max-results SHIPPED (2026-08-20). The aggregate ceiling applies to the implicit result
    listing after the complete expression, across branches and independent caps. It does not stop the
    walk or truncate reductions, so summaries, histograms, counts, and packing remain complete.
    Explicit actions retain expression semantics: a global cannot retrospectively suppress -print
    or -grep records already emitted while deciding the expression, and -exec is not a listing;
    put -first / -top before an action to cap what reaches it. Last occurrence wins, zero lists
    none, and missing / malformed / negative counts are pre-walk usage errors. End-to-end coverage
    pins the 10-files-or-5-directories capped-to-12 case, complete reductions, explicit actions, last
    wins, zero, and every invalid form.

Silence external warnings in the EXEC configuration too (from mboworks/mbo#332) - SHIPPED

Adapted from mbo's #332, which found that the warning
policy applied only to the TARGET configuration, so anything built for the host escaped it in both
directions: first-party warnings were not errors, and external sources were not muted.

Two changes, and the first is the one that mattered:

The carve-out is the part that is NOT a copy of mbo. The composable extras are FIRST-party code
that happens to live under external/ (separate modules via local_path_override), so both mute
rules keep ,-external/xff_.* and both configurations get an explicit
external/xff_.*@-Wextra,-Wpedantic,... rule. mbo's blanket external/.*@-w would have silently
muted every warning in @xff_archive, @xff_pcre2 and @xff_fuse.

Verified by planting a deliberate unused variable and building, rather than by reading the flags:

Where Result
xff/fuzzy/fuzzy.cc (first-party) error: -Werror, fails
external/xff_archive+/archive_reader.cc error: -Werror, fails
third-party external sources 0 warnings (was: on every build)

Resolved (2026-08-22): eliminate the compile-DB extractor's C/C++ driver mismatch. The
PR #611 clang-tidy job
again prints ~80 error: invalid argument '-std=c99' not allowed with 'C++' diagnostics. The
extractor probed third-party C targets (xz sets -std=c99) with clang++; warning flags could not
affect that language-mode error, and dropping those third-party entries only cleaned the resulting
database, not the refresh log. compile_commands-update.sh now names the hermetic, language-neutral
clang driver: the source extension and extracted flags select C versus C++, while clang-tidy can
still introspect the real driver for its target and resource directory. The output filter was
removed; the third-party C post-processing and first-party coverage guard remain.

Compile-database coverage guard (clang-tidy cannot lint what the database omits)

clang-tidy lints only the files the database LISTS, so a source bazel compiles but the extractor
missed is silently unlinted, and the run still looks clean. That is not hypothetical: every extras
translation unit was absent once (the story is in //:refresh_compile_commands).

compile_commands-update.sh now asks bazel which first-party .cc files are in a cc_* rule
(//xff/..., the derived extras wildcards, @xff_extras_api//...) and fails if any lacks a database
entry, naming the files. Bazel's query is the authority on "compiled" rather than a find over the
tree - a file in no target is a different problem.

Two properties the implementation must keep, both learned the hard way while writing it:

Current state: 165 first-party sources expected, 0 missing. Verified the guard FAILS (exit 1, naming
the file) on a source that has no entry.

squashfs: bottom of the stack, read first (decided 2026-08-18; reader shipped 2026-08-28)

Build it when a concrete "search inside a snap / AppImage / firmware image" need appears, not for
format completeness. Three parts in order: READ without lzo, then WRITE (create from a walk, then
repack for modification), then lzo if ever.

Structurally it is another container on the existing VFS seam and needs no FUSE at all - diving with
--archive never mounts anything - and it is in some ways nicer than tar, because it has a real
index (indexed directory lookup, indexed seeking within a file). Writing is multi-pass with
backpatching, which is the ordinary shape here (buffered tabular renderers, --sort=score,
-collect, the -top design). Two real constraints on writing: the output must be SEEKABLE, so
packing to a pipe gets refused, and buffering the tree needs the bounded-memory treatment -collect
has. Shipping without block dedup and tail-packing into fragments is valid but produces bigger images
than mksquashfs - a quality gap, not a correctness one.

The licensing is the load-bearing part, verified 2026-08-18:

The shipped slice is @xff_squashfs, gated by --//xff:xff_squashfs and composed by
--//xff:xff_all. It registers independently through @xff_extras_api, so enabling it does not pull
in libarchive. Raw images, Snaps, and AppImages become ordinary virtual member trees and use the same
predicates, fields, content reads, nesting, and summaries as other containers. AppImage's executable
prefix uses libsqsh's archive-offset API after a bounded scan for the hsqs superblock. A committed
fixture and constructed prefixed variants cover path-backed and retained-byte inputs. Creation and
rewrite remain later work because a correct SquashFS writer is separate, multi-pass work.

Bashtest scratch files: use test_tmpdir, not hand-rolled paths

mboworks/bashtest already provides ${BASHTEST_TMPDIR}, a scratch directory its own exit trap
removes, and 19 xff bashtests ignored it to hand-roll mktemp -d plus a per-case rm -rf. Beyond
the duplication, the cleanup is WRONG in the case that matters: bashtest keeps running after a failed
expectation, so a case that fails before its rm leaks its tree.

Bashtest now owns per-fixture allocation too. The convention is:

_tree() {
  local root
  root="$(test_tmpdir tree)"
  # Build the fixture under ${root}.
  echo "${root}"
}

root="$(_tree)"

${FUNCNAME[1]} (the calling test) was tried first and is wrong: the test's name then appears in
printed paths, where it can satisfy an expect_output_not_contains. That is not hypothetical -
archive_test's "archive is off by default" case failed exactly that way. test_tmpdir is unique per
call, so a case that builds two trees needs nothing special and no mutable counter. Per
STYLE_SH.md, a helper returns the path through stdout (root="$(_tree)") rather than
mutating a caller-named variable with printf -v.

Complete: all 19 converted. archive_test, color_test, parity_test, first_test, ignore_test, collect_test,
content_test, fuzzy_test, grep_test, ls_test, exact_test, pager_test, cmp_test,
full_binary_test, ignore_files_test, help_topic_test, archive_pack_test, archive_dive_test,
summary_test, ignore_gitignore_test - 19 of 19, with 214 rm -rf lines gone. The later repository-wide audit
also migrated every other bashtest-created directory, including diff/hash/histogram/csv fixtures,
man-page scratch, style/exit/exec helpers, and the cookbook's shared fixtures: 114 allocations across
33 test files now go through test_tmpdir. tools/release_prep_test.sh is the deliberate exception:
it is a standalone pre-commit test, not a bashtest, and owns its temporary repositories and cleanup.

The location-sensitive archive, summary, and git-ignore assertions pass unchanged; the conversion
needed fixture-lifetime work, not weakened matching. Unique allocation handles several fixtures per
case with no special treatment and avoids caller mutation, collisions, and test-name leakage.

libfuse: no build dependency at all (SHIPPED 2026-08-18)

The FUSE extra used to compile against libfuse's fetched headers, which put xff in the position of
ARGUING that LGPL-2.1 section 5 permits it. The argument was sound - data structure layouts and
accessors are outside that section's restrictions, and its ten-line limit applies only to inline
functions - but xff ships statically linked single-file binaries, so the whole question was load
bearing on a reading of someone else's licence.

It is gone. extra_modules/fuse/fuse_abi.h declares the fuse3 lowlevel ABI itself, the
http_archive is commented out (kept as provenance and as a conformance tool: uncomment it plus the
@libfuse//:fuse3_headers deps to diff our declarations against the real ones after a libfuse
release), and no libfuse file is fetched, read, compiled, linked or shipped. The runtime library is
unchanged: fuse_loader dlopens whatever the host has.

Fidelity rules the header follows, since an ABI description is only worth its accuracy:

How a mistake surfaces: the loader resolves every entry point by NAME, so a wrong signature or
offset cannot fail at link time - it corrupts at runtime. Linux CI is the guard, where
fuse_server_test and the --archive-mount CLI cases drive a real fuse3.

Notice correction. The intermediate empty-SPDX interoperability entry was also superseded. The
final model is recorded below: the component is xff's own Apache-2.0 FUSE extension, while the
runtime-provided libfuse/macFUSE implementation is not a component of the binary.

libfuse: no build dependency, and the notice lists OUR code (SHIPPED 2026-08-18)

The FUSE extra used to compile against libfuse's fetched headers, which put xff in the position of
ARGUING that LGPL-2.1 section 5 permits it. The argument was sound - data structure layouts and
accessors sit outside that section's restrictions, and its ten-line limit applies only to inline
functions - but xff ships statically linked single-file binaries, so the whole question was
load-bearing on a reading of someone else's licence.

extra_modules/fuse/fuse_abi.h now declares the fuse3 lowlevel ABI itself. The http_archive is
commented out (kept as provenance and as a conformance tool: uncomment it plus the
@libfuse//:fuse3_headers deps to diff our declarations against the real ones after a libfuse
release), and no libfuse file is fetched, read, compiled, linked or shipped. fuse_loader still
dlopens whatever the host has.

Fidelity rules the header follows, because an ABI description is only worth its accuracy:

How a mistake surfaces: the loader resolves entry points by NAME, so a wrong signature or offset
cannot fail at link time - it corrupts at runtime. Linux CI is the guard, where fuse_server_test and
the --archive-mount CLI cases drive a real fuse3.

The notice model, corrected twice before it was right. The old entry claimed
libfuse [LGPL-2.1-only] and "compiled against its headers". That misstated the project (LGPL-2.1
covers include/, lib/, meson.build; everything else is GPL-2.0), asserted -only more
firmly than the LICENSE file supports, and named the runtime library wrongly - on macOS it is macFUSE,
under its own terms. An empty SPDX was tried next and is also wrong: [] reads as public domain.

The rule that settles it: list what you USE, and our own code is Apache-2.0. So the registered
component is xff FUSE extra (@xff_fuse) with SPDX Apache-2.0, whose text describes the libfuse
interaction as INFORMATION - our own declarations, host implementation loaded at runtime - and libfuse
is not registered at all, because none of its code is in the binary. A test pins both halves.

Consistency follow-up (SHIPPED, PR #569): @xff_archive and @xff_pcre2 now register their own
Apache-2.0 extension entries before the third-party components they link, matching @xff_fuse and
making the manifest complete and uniform.