Intro

All contributions are generally welcome as long as they fit in with the concepts and goals of this repository and as long as the CODE_OF_CONDUCT.md is being respected.

Code Rules

All code must adhere to the RULES.md and mostly follows the Google style. Where it diverges, clang-tidy rules are in effect as much as possible.

Coverage

The production coverage job derives its ratings, HTML colours, and CI gates from
coverage_policy.json. See docs/coverage.md before
changing thresholds or adding a category override; weakened overrides require a documented reason.

Run pre-commit

All changes will be verified by the pre-commit rules. In order to check these before committing changes install the tool:

pre-commit install --hook-type pre-commit --hook-type pre-push

Once installed the verification can be triggered for all files as follows:

pre-commit run -a

Without the -a only the modified and staged files will be checked.

Python tooling and test ownership

Use Aspect py_library, py_binary, and py_test from
@aspect_rules_py//py:defs.bzl. MODULE.bazel pins the rules and selects the standalone
Python 3.13 toolchain supplied by rules_python; Bazel execution does not use ambient python3.
Declare imported modules as dependencies and non-Python inputs as runfiles, with narrow visibility.
A filegroup describes data, not an executable Python program.

Run all 34 tooling suites with bazel test //tools:python_tests; they also run under bazel test //....
The benchmark integration is //xff/engine:read_benchmark_test. Run the measurement commands with
bazel run //tools:measure_resources -- -- COMMAND... or
bazel run //tools:benchmark_resources -- XFF --output=REPORT.json.
The benchmark driver declares its measurement executable and runs each measurement in a fresh
process, preserving per-invocation peak-memory accounting. Direct script invocation remains available.

Pre-commit owns changed-file formatting and policy checks, plus fast, narrowly triggered checker
self-tests. Its Python hooks use the managed Python 3.13 environment. Process-measurement and fuzz
scheduler suites belong to Bazel, so commits do not run those suites or start recursive Bazel builds.
Main and release coverage CI run these suites through their existing Bazel test/coverage commands,
without additional direct unit-test invocations. The Pages workflow is a compatibility exception:
it checks out historical release tags and tests that tag's site builder directly, since older tags
need not provide Bazel Python targets.

bazel run //tools:repository_tooling_test is the repository-wide integration check. It uses
BUILD_WORKSPACE_DIRECTORY to inspect every BUILD file, the extra-module registry, and test ownership
in the live checkout. Main and release CI run it explicitly. It is a py_binary using the same
hermetic interpreter, not a sandboxed or cached py_test; its assertions execute on every run.
No list of exported BUILD files or source snapshots needs maintenance. For direct script execution,
supply --workspace=PATH explicitly. Fixture-based unit suites remain under bazel test.
The coverage-index and release-site suites require host Git; release-note tests require Bash and sed.
Their Bazel targets carry requires-host-tools, and the supported Linux/macOS CI images provide them.
No tests contact GitHub or recursively invoke Bazel; the separate site integration step does use GitHub.

Local clang-tidy scope

The normal commit and push hooks run clang-tidy on their changed C++ files. Changed headers
select the affected translation units. A single coordinator defaults to the detected CPU count minus one
(minimum one), capped by the number of selected files. This leaves one CPU available for other work. Set CLANG_TIDY_JOBS to a positive integer to override
that count, or leave it unset (auto) to use CPU detection. GitHub CI explicitly supplies its
runner CPU count; there is no fixed two-worker local limit.

When Trunk manages Git hooks, the repository's Trunk actions invoke this same pre-commit hook.
Both Trunk actions explicitly forward CLANG_TIDY_JOBS, so a per-command worker override reaches
the coordinator (for example CLANG_TIDY_JOBS=8 git push).
The push action combines changed paths from the pushed refs into one invocation. Existing Trunk
formatting and checks remain enabled; do not replace core.hooksPath to install competing hooks.

Generate or refresh the compile database with ./compile_commands-update.sh, including after
adding sources or changing build configuration. Without the database or clang-tidy 22 or newer,
the local hook reports a skip; the dedicated CI job supplies both and enforces the check. A requested
source missing from an existing database fails with a refresh instruction instead of silently
omitting that file.

Check selected files, including previously committed files that failed CI:

pre-commit run clang-tidy --files xff_extras_api/mutations_test.cc xff_extras_api/fuse_backend_test.cc

A full scan is a separate, explicit invocation (not part of a normal commit or push):

pre-commit run clang-tidy --all-files --hook-stage manual

As with other hooks, pre-commit run -a explicitly selects all files. Use --files for a focused
verification; omit it to check staged changes.

Generated dependency sources needed during compile-command extraction must be explicit build
outputs in the owning extra. The PCRE2 extra exposes chartables_source for this purpose;
@xff_pcre2//:chartables_test compares it byte-for-byte with PCRE2's versioned
pcre2_chartables.c.dist. The compile-database preparation builds the extra's wildcard targets,
so the generated source is requested even when the compiled library is cached.

CI build caches

Measured cache reuse, scheduling, fuzz throughput, and preparation profiling are documented in
docs/ci-performance.md.

Coverage and fuzz jobs start alongside pre-commit and Trunk to reduce total CI wall time.
The final done gate still requires every job to pass. This intentionally spends runner time
on those jobs even when an early lint check fails.

Bazel's compiled-output disk cache is separate from its repository download cache.
CI restores the latest main cache for each runner OS, architecture, and build configuration.
Only main writes refreshed generations; PRs and release tags restore without saving. GCC also
uses a compiled-output cache. Release builds share the production configuration's main cache.

GitHub cache entries are immutable. Every main run therefore saves under a new run/attempt key,
with prefix restoration to reuse the previous generation. During migration, existing legacy
compiled-output caches remain a fallback until the first new generation is saved. A GitHub "cache hit" only means the
archive was restored; Bazel's final process statistics show whether build actions actually hit
that cache. Dependency and compiler-option changes can still require rebuilding.

Before upload, CI stops Bazel and evicts the largest disk-cache files until the action-cache and
content-store payload is at most 1,000,000,000 bytes by default. ASan uses 3,000,000,000 bytes and TSan uses
2,600,000,000 bytes; MSan uses 4,000,000,000 bytes and coverage uses 1,500,000,000 bytes
to retain more instrumented outputs. These are uncompressed limits; GitHub
storage usage reflects compressed uploads. Measure those uploads and total repository usage
after main refreshes the caches before increasing other configurations. This is a synchronous upload bound, not an
idle-time garbage-collection setting. Evicted outputs become normal cache misses; this bounded
cache does not promise to retain every object or executable. Size-first eviction favors retaining
many smaller compilation outputs over large linked executables; ties evict older files first.
This is a size heuristic, not action-type classification: large object files can also be evicted.
The weekly deep-fuzz workflow shares the ordinary fuzz cache and main-only refresh policy.
LLVM downloads remain uncached.
Cleanup retains ASan/TSan/coverage uploads up to 1,000,000,000 compressed bytes and MSan uploads up to
1,200,000,000 bytes; other entries keep
the existing 700,000,000-byte ceiling. The repository storage limit remains 10 GB.

After each successful main upload, the save action checks the GitHub inventory for that exact
replacement key and its compressed size. Only then does it delete strictly older main generations
for the same OS, architecture, and configuration. Missing, empty, or oversized replacements leave
the previous generations intact; concurrent newer uploads are also preserved. Inventory and
deletion failures leave cleanup to the fallback workflow. Compiler jobs need Actions write
permission for deletion; PRs and tags never invoke the save/retirement action.

The trusted cache-maintenance workflow removes closed-PR and tag-scoped caches first, then
superseded generations per configuration and ref, and oversized entries. It retains the newest
usable generation if a newer upload exceeds the ceiling. Open-PR caches from older workflows are
not removed merely because they belong to a PR; new PR runs do not write such caches.

Main's done job reports each measured cache's pre-trim and retained uncompressed sizes,
compressed upload size, and eviction count. It also totals all repository cache entries against
the 10 GB budget, including any overlapping generations still present. Missing uploads or job
measurements are identified rather than reported as zero. Small measurement artifacts expire
after one day and are not part of the cache-storage total.

Locally, use consistent compiler, extras, and linker options when comparing incremental builds.
An analysis-cache reset alone does not mean compilation was repeated. A personal --disk_cache
setting can preserve action outputs across configuration switches and checkouts; the CI cache
change does not alter local Bazel settings.

Parallel fuzz campaigns

tools/fuzz_targets.py --campaign-seconds=60 discovers every first-party campaign, builds all
launchers together, and prepares Bazel run scripts sequentially. It then executes those scripts
concurrently, avoiding competing Bazel invocations. CI retains 60 seconds per target; weekly/manual
Deep fuzz uses 600 seconds per target with a 7,200-second aggregate budget.
--max-total-seconds still limits the sum of all target budgets,
not parallel wall time.

Local concurrency defaults to CPU count minus one. --jobs=N selects a maximum; both CPU capacity
and physical RAM further bound it. The driver reserves 2 GiB for the host and 512 MiB of overhead
per worker beyond --rss-limit-mb (default 2048 MiB). CI requests the runner CPU count through the
same bounded scheduler. Unknown physical RAM falls back to one worker. The estimate does not
measure other processes' live memory use; lower --jobs on a busy machine.

Each campaign has its own output directory, log, corpus, crash artifacts, and result JSON.
--output-dir=PATH chooses the parent; otherwise the driver creates and retains a temporary
directory and prints its path. Generated launch scripts establish the runfiles context. On failure,
the scheduler stops queued campaigns and terminates active campaign process groups. A campaign
that exceeds its budget by 60 seconds is terminated as a failure, covering hung launchers or inputs.

The fuzz-campaigns CI artifact retains logs and findings for seven days, even after failure.
summary.json records workers, campaign wall time, and per-target execution counts and average
executions per second when libFuzzer supplies them. Compare those statistics alongside elapsed time:
CPU or memory contention can reduce mutation throughput even when parallel wall time improves.

Release-artifact smoke checks

The default Linux/macOS jobs and tag builds run tools/release_smoke.py against both
stripped executables after staging. The disposable corpus checks the exact stamped version,
help, JSONL/CSV/NUL filename round trips, escaped plain/TSV record boundaries, named INI
selection, repeated summary identity,
a literal total extension, and all four comparison outcomes. No installed executable or
user data is modified. Existing required system/user configuration still applies; run this
release check in a clean CI environment. It complements the full tests rather than replacing them.

To check an already built development binary:

python3 tools/release_smoke.py PATH_TO_XFF --expected-version=0.0.0

Use the tag's numeric version for release binaries. Each invocation has a 30-second timeout,
and a failure prevents artifact publication.