xff - eXtended File Find

Release website

xff is a find(1)-compatible file finder with modern extensions. It walks each starting path and acts on entries through find's expression grammar, then adds content and language search, structured output, per-run summaries and histograms, safe deletes, native hashing, and a shared {field} vocabulary that threads through -printf, -exec, and every renderer.

The find style restricts the expression to find-compatible primaries, operators,
and values. Whole-run xff globals remain available when explicitly supplied, while
the default xff style intentionally selects modern presentation and traversal
defaults.

This README.md is a short overview. The complete, always-current reference lives in XFF.md (generated from the binary; see Documentation).

Quick Start

Download the executable for your platform from the latest release.
Choose xff for the lean core, or xff_full for all extras, including PCRE2 and archive support.
The release includes checksums, verification instructions, and separate archives with debug symbols.
You do not need Bazel to use a published executable.

For example, after downloading xff-macos-arm64:

chmod +x xff-macos-arm64
./xff-macos-arm64 . -type f -name '*.md'

Place the chosen executable on your PATH as xff to use these commands:

# Find source files.
xff . -type f -name '*.cc'

# Search file contents.
xff . -type f -grep 'TODO'

# Compare two trees without listing every result.
xff --compare=summary LEFT RIGHT

# Report matched file counts and apparent bytes by extension.
xff . -type f --summary=ext

Summary sizes are apparent entry sizes, not allocated disk blocks or recursive directory sizes.
For building from source, see Building & Dependencies.

Core Highlights & Architectural Advantages

• find-Compatible Core: The standard primaries (-name, -type, -size, -mtime, -regex, -exec, -prune, ...), operators, and exit-status conventions provide a portable union of GNU, BSD, and POSIX find behavior. Invoking the binary as find selects the find expression vocabulary and compatibility-oriented defaults while retaining explicit xff global controls.
• Content & Metadata Matching: -grep / -content search inside files, -lang 'C*' and -mime 'image/*' match by inferred language or media type, -text / -binary / -eofnl classify content, and native -hash primitives emit optimized checksum manifests.
• Overrideable MIME Vocabulary: The lean binary carries common media types; the removable
  mime-db extra expands that to thousands of registered types. Repeatable
  --mime-vocabulary=FILE JSON overlays can replace extension mappings and attach descriptions,
  sources, charsets, aliases, and compressibility, with explicit conflict policy and matching
  {mime-*} fields for output and aggregation.
• Overrideable Language Vocabulary: Common languages remain built in; the removable,
  Brotli-compressed GitHub Linguist extra expands classification to hundreds of canonical records.
  Repeatable --lang-db=FILE layers override suffixes, exact filenames, aliases, colours,
  groups, and provenance with an explicit ambiguity policy; {lang-*} fields expose the selected
  metadata without running a heavyweight content classifier, and the same colours theme regular
  filenames consistently in the plain and -ls listings.
• One Composable Expression Language: Path, content, ownership, permissions, age, allocated size, language, MIME type, hashes, and content equality are ordinary tests joined with find's !, -a, -o, and parentheses. Search, reporting, and actions therefore share one walk instead of being stitched together with xargs and temporary files.
• Fuzzy File Finding: -fuzzy and -fuzzypath provide scored fzf-query, plain-subsequence, Levenshtein/edit, and character-shingle models, while --sort=score ranks the results. The score is also available as {fuzzy} for custom tables and templates.
• Structured Layout Engines: Stream matches natively as plain text, NUL-delimited, JSONL, CSV / TSV, an aligned console table, a visual tree, or a standard Markdown table, all calculated from one single filesystem walk.
• Summaries & Histograms: --summary=ext folds matches into counts, sizes, and percentages, with combined, per-root, or paired comparison tables (see summary scopes); --histogram='ext:sum(lines)' draws terminal bar charts using Unicode block characters. No external awk | sort pipeline overhead required.
• Native Comparison & Deduplication: --compare[=status|diff] LEFT RIGHT [EXPRESSION] applies the requested traversal, symlink, ignore, and expression semantics independently to both roots, then reports selected result kinds or a unified patch; the one-sided -diff TARGET action compares every match from one walk with its templated counterpart. -cmp tests byte equality, -hasheq verifies a digest, and --summary=hash groups duplicate content.
• Near-Duplicate Content Matching: -similar compares every matched text file with a reference using exact word-shingle Jaccard similarity. Its shingle width and quality threshold are explicit, so source forks, copied documentation, and lightly edited configuration files can be found without reducing the question to byte equality.
• Unified {field} Vocabulary: The same named fields ({relpath}, {size}, {lang}, {hash}, {capture.NAME}, ...) drive -printf, -exec, --format, and --summary, complete with powerful s/// regex rewrite and m// extraction qualifiers.
• Commands as Data Sources: -capture / -capturedir bind a command's output to {capture.NAME}. Later formatters and reductions can transform or aggregate it, enabling workflows such as tree-wide git blame totals without an awk | sort tail.
• Configurable Safety: unrestricted by default, with an adjustable --safe profile, mandatory --block-* restrictions, and --dry-run previews. Each INI file can distinguish archive operations and recursive temp/output-directory permissions from ordinary file operations. Explicit and autoloaded .xffrc files cannot arm their own execution or deletion actions; trusted --allow-exec arming never bypasses operation blocks. See --help=safety for the operation table and limits.
• Multi-Threaded Traversal: -j N uses a native worker pool for directory reads and independently permits up to N concurrent -exec jobs, scaling both discovery and actions across available CPU cores; --sort restores deterministic ordering when requested.
• Virtual Archive Filesystems: --archive walks archives and compressed files as directory trees, including nested containers, Electron ASAR bundles, SquashFS images, Snap packages, AppImages, and Apple PKG/XIP application payloads on Linux and macOS. The normal name, metadata, content, hash, and reduction vocabulary works on members; removable format extras stay independently linkable rather than pulling every archive dependency into the core. Opt-in controls can mount or extract members for commands and safely rewrite formats that support deletion.
• Archive Creation from an Expression: --pack=FILE writes the matched set directly to a new archive, with deterministic member order, atomic publication, compression controls, and no find | tar filename boundary to get wrong. The removable Brotli extra reads raw streams and writes self-identifying RFC 9841 .tar.br / .tbr by default, with an explicit raw compatibility mode.
• Shared INI Configuration: System, user, and .xffrc files share CLI flag syntax and composable named profiles. Project autoloading is opt-in with --rc or recursive --rc+; unsectioned project settings require a trusted grant and otherwise fail before actions run. See configuration.
• Developer-Aware Traversal: Layered .gitignore, .ignore, and .xffignore handling; explicit include/exclude rules; hidden-file policy; and pruning for Git, Mercurial, Subversion, Jujutsu, Bazaar, Darcs, and CVS are independent, configurable controls.
• Shard-Aware Validation: --shards recognizes numbered datasets such as data-00000-of-00010
  and collapses each set to a useful first, wildcard, or count representation;
  -shard-status complete|incomplete|superfluous selects healthy sets, missing-index sets, duplicate
  copies, and declared-total outliers for inspection or action.

Tool Feature Comparison Matrix

The matrix compares native, built-in capabilities: ● means supported, ◐ means partial support,
and an empty cell means no native support (another utility or a shell pipeline is normally needed).
Partial support covers a narrower form of the feature. The point is not that every specialist is
interchangeable, but that xff composes these operations in one expression and one traversal.

find's "Field templates and rewrites" entry is marked GNU because it is GNU find's -printf, a
GNU extension; POSIX and BSD/macOS find have no format primary (only -print / -exec).

Flavor & Behavioral Shift Matrix

xff runs one unified grammar under three operational flavors. The flavor is selected automatically by the program binary name and can be explicitly overridden or layered using the --config flag (where the last specified style wins).

The table below illustrates how traditional shell workflows shift into optimized xff unified expressions.

* The find style is still xff's engine, not a wrapper around the OS find. It restricts
the expression vocabulary, but xff globals such as --format, --sort, and --jobs remain
available as explicit overrides. The clearest divergence is regex: -regex / -iregex
default to RE2 (linear-time, no catastrophic backtracking) and behave identically on Linux
and macOS - where GNU find instead defaults to its Emacs dialect and BSD/macOS find to BRE.
-regextype selects xff's grammar set (RE2, ERE, EXACT, FNMATCH, GLOB, SHGLOB, plus PCRE2 in
a full build), never GNU's dialect names. ERE deliberately uses the platform POSIX implementation;
the other core grammars are cross-platform. GLOB and SHGLOB are locale-independent, component-aware,
and reject malformed bracket expressions instead of silently changing their meaning. SHGLOB adds
nested alternatives and bounded integer or ASCII-letter sequences such as {01..12} and {a..z}.
Otherwise the find expression vocabulary follows the documented GNU, BSD, and POSIX behavior that xff makes uniform across platforms.

Examples

# Ten largest files (-printf builds any columnar line; the shell sorts).
xff . -type f -printf '%s\t%p\n' | sort -rn | head

# Apparent bytes per extension (--summary may sit at the end).
xff . -type f --summary=ext

# Delete stale temp files, safely (prints what -delete WOULD remove).
xff . -type f -name '*.tmp' -mtime +7 -delete --dry-run

# Search code content, filtered by language (path:lineno:text for every TODO).
xff src -lang 'C*' -grep 'TODO'

# Checksum manifest for a tree (like sha256sum: `DIGEST  PATH` per file).
xff . -type f -hash:sha256

# Recently changed files as machine rows (one JSON object per file, for jq).
xff . -type f -mtime -1 --format=jsonl

See the XFF.md cookbook for more worked examples, including native per-author git blame line counts computed with no shell pipes.

Documentation

The vocabulary and options are defined once inside the C++ binary (the engine registry acts as the single source of truth), ensuring that every documentation surface is automatically generated and cannot drift:

• XFF.md: The full comprehensive reference in Markdown. Run xff-md-update.sh to regenerate it; the //xff/cli:xff_markdown_test target fails CI if it drifts from the binary.
• xff --help: Renders the main utility usage page. Use xff --help=TOPIC to review specific sub-topics (fields, printf, time, size, grammars, stats, etc.), or xff --help=full to dump all help sections.
• Formatted full reference: Append :markdown (or :md), :html, or :roff to --help=full (or its long alias). HTML is a standalone, self-contained HTML5 document with no scripts or external assets; --man remains the conventional alias for the roff form.
• Order and limits: Choose expression filters, display limits, or traversal termination, and see how each affects actions, reductions, and packing.
• Paired release docs: The shared //xff/cli:xff_reference_gen action always generates Markdown and HTML together, and release Pages publish that exact pair so the formats cannot silently diverge.

Building & Dependencies

Building from source requires Bazel:

# Build and run the lean binary.
bazel run //xff -- . -type f -name '*.md'

# Or build once; the executable is bazel-bin/xff/cli/xff.
bazel build //xff

The default build provides a lean, dependency-light core. Heavier processing capabilities (such as the advanced PCRE2 regex grammar or recursive archive diving) are decoupled as composable build-time extras that are disabled by default to keep the core binary small. The extended target links them all:

# The full binary, with every extra (PCRE2, archive diving, Brotli, etc.).
bazel build --config=xff_full //xff/cli:xff_full

The //xff target alias follows your active workspace configuration automatically: it resolves to the lean binary by default, and switches to the full binary under --config=xff_full. The underlying targets remain explicit and configuration-stable: //xff/cli:xff is always lean, and //xff/cli:xff_full is always full.

Published binaries use --config=clang_release, which combines the hermetic Clang toolchain with
size optimization and ThinLTO. Releases provide the stripped xff-PLATFORM-ARCH and
xff_full-PLATFORM-ARCH executables directly. Each platform also provides separate level-19
Zstandard-compressed xff-PLATFORM-ARCH.tar.zst and xff_full-PLATFORM-ARCH.tar.zst archives.
Each archive contains its own executable and matching symbol file under debug/; no .tar.gz
duplicate is published. See the latest release
for downloads and installation instructions. The ordinary Linux and
macOS CI jobs run tagged binary-level tests with this same configuration and execute the staged,
stripped binaries before a release can use it; the full unit-test graph stays on the same hermetic
Clang toolchain without paying ThinLTO cost for every test executable. Each release also publishes
an attested SHA256SUMS manifest and installation instructions, including generation of the
matching man page directly from the installed binary with xff --pager=never --man.

Compile-Time Enforcement: The CLI options for extras (e.g., --regextype=PCRE2 or --archive) are always exposed on the interface. Attempting to invoke an extra feature in a lean build that did not compile it will yield an immediate, explicit error rather than a silent failure or fallback.

• Requirements: Bazel 9.1.1 or newer, accompanied by a modern C++23 toolchain (clang-22 or
  newer). A fully hermetic LLVM toolchain is available out-of-the-box via --config=clang.

Contributing

See CONTRIBUTING.md, the LLM agent and contributor guidelines in AGENTS.md, and the coding styles in STYLE_CPP.md and STYLE_SH.md. In-depth design notes live under docs/; current work is tracked in TODO.md, while completed investigations and decisions move to the development history.

License

Copyright M. Boerger, the MBO Works authors. Licensed under the Apache License 2.0; see
LICENSE, Lean build notices, and
Full build notices for details. For custom builds, --help=notice lists
exactly the components included in the running binary.