| name | tensor-grep-architecture-contract |
| description | Use when you need the load-bearing design of tensor-grep and WHY it holds before touching cli/bootstrap.py, rust_core/src/main.rs, backends/, core/result.py, cli/main.py's native-delegation gate, routing, the agent capsule, or before reviewing/planning any change to the front door, command/flag registration, or backend contract. Explains the bootstrap intercept-before-Typer front door, native-vs-Python routing, the 4 command + 2 flag registration sites plus the MCP contract-version site, the Backend Fail-Closed Contract, the native-delegation forward-or-refuse contract (`_can_delegate_to_native_tg_search` + its field-coverage ratchet), the partial-results `result_incomplete`/`incomplete_reason` envelope, `MatchLine`'s frozen-but-hashable dataclass contract, the ASCII-only CLI output rule, the agent-context moat, the invariants that must hold, and the known-weak points (flat no-IDF scorer, GPU not viable, rg parity gap, FFI not the dir-scan speed path). Read this to build the right mental model; use sibling skills for the how-to of changing, debugging, or benchmarking. |
tensor-grep architecture contract
What this is. A ground-truthed map of tensor-grep's load-bearing design: the invariants a change must not break, and the weak points you must not oversell. Read it to understand why the code is shaped this way before you touch it. It is not a how-to — for that, hand off to a sibling (routing table below).
What tensor-grep is (as of 2026-07-24, v1.95.0, pyproject.toml): a code-intelligence CLI named tg. A Rust core (rust_core/ — both a PyO3 extension and a standalone tg binary) plus a Python CLI (src/tensor_grep/). Apache-2.0. Ships to PyPI (package tensor-grep), npm, Homebrew, winget. CONTRIBUTING.md calls it a "benchmark-governed, contract-heavy codebase" — that is the whole point: the contracts below are enforced by tests and a CI gate, not by convention.
When to use this skill vs a sibling
| You are about to… | Use |
|---|
| Understand why the front door / routing / backend contract exists (this skill) | you are here |
| Add/rename a command or a search flag; ship a change safely | tensor-grep-change-control |
| Debug a live misroute, hang, wrong-result, or "no matches that should match" | tensor-grep-debugging-playbook |
| Study a settled past failure so you don't re-fight it | tensor-grep-failure-archaeology |
Use tg as a user (search/orient/callers/agent flags) | code-search-and-retrieval-reference, or the .claude/skills/tensor-grep/ usage skill |
| Set/override config or env axes | tensor-grep-config-and-flags |
| Build the Rust ext / set up the toolchain | tensor-grep-build-and-env |
Run diagnostics (doctor, dogfood, readiness) | tensor-grep-diagnostics-and-tooling |
| Make or defend a speed/quality claim with numbers | tensor-grep-benchmark-and-proof-toolkit |
| Position the product / write release notes | tensor-grep-release-and-positioning |
Do not use this skill to authorize a change. It explains the design; it does not route around tensor-grep-change-control or the project's PR/council/dogfood discipline. Any code change still goes through change-control.
Jargon (defined once)
- Front door / bootstrap — the process entry point
tensor_grep.cli.bootstrap:main_entry that sees raw argv before the Typer app.
- Typer app — the Python click/Typer CLI in
src/tensor_grep/cli/main.py (@app.command functions). It is the inner CLI, not the front door.
- Native binary / native front door — the standalone Rust
tg binary built from rust_core/. Fast path for search routing.
- Sidecar — Python doing work the native binary bounces to it (
TG_SIDECAR_PYTHON).
- CliRunner — Typer's in-process test harness. It calls the Typer app directly and bypasses the bootstrap front door — the single most important test-coverage caveat in this repo.
- rg — ripgrep. ast-grep — structural (AST) search. Both are baselines tg is measured against, not beaten.
- Capsule — the Actionable Context Capsule emitted by
tg agent (capsule_version = 1).
The front door: intercept before Typer
tg is not "a Typer app." The published entry point is bootstrap.main_entry — grep -n "^def main_entry" src/tensor_grep/cli/bootstrap.py (was :1444, now :1550; this anchor has drifted three times, never cite the number as fact). It parses argv itself and, for a plain text search, forwards to the native tg binary or to ripgrep before Typer ever runs (re-grep _run_rg_passthrough — the dispatch runs from the _normalize_search_invocation call through the final raise SystemExit(_run_rg_passthrough(...)); traced live 2026-07-29 and confirmed a bare tg search PAT exits THERE, never reaching Typer). The Typer app is only reached for TG-only flags, help, or commands that require full CLI (grep -n "^def _requires_full_cli" src/tensor_grep/cli/bootstrap.py — :480, unchanged this pass).
Why this matters, concretely:
- CliRunner cannot see routing bugs. It invokes the Typer app directly, so any bug in
bootstrap routing (a flag that leaks to rg, a fork-bomb delegation loop, a wrong native/Python choice) is invisible to CliRunner tests and green in CI while broken for real users. This is exactly how the --rank plain-text crash shipped (AGENTS.md §"Dogfood the Real Binary, Not CliRunner" — grep -n "^## Dogfood the Real Binary" — was :422, now :957). Rule: verify front-door behavior against the REAL published binary via scripts/dogfood/ (Dockerfile + dogfood_features.py), never CliRunner alone.
- Two mutual-delegation fork-bomb hazards are guarded, not theoretical.
TG_REEXEC_GUARD (grep -n 'os.environ.get("TG_REEXEC_GUARD")' src/tensor_grep/cli/bootstrap.py — was :1497, now :1617) stops native→python→native search loops; _json_aggregate_blocks_passthrough (grep -n "^def _json_aggregate_blocks_passthrough" src/tensor_grep/cli/bootstrap.py — :572, unchanged this pass) stops --json + a render-only flag (e.g. -b) from deadlocking the native front door; _run_requires_ast_workflow (grep -n "^def _run_requires_ast_workflow" src/tensor_grep/cli/bootstrap.py — was :1385, now :1491) keeps tg run --selector/--strictness/--stdin/--globs in Python so it does not ping-pong. If you touch delegation, you can re-arm a fork bomb — see tensor-grep-failure-archaeology.
Native-vs-Python routing (the decision tree)
Search routing is a single shared decision in rust_core/src/routing.rs::route_search(...) (documented in docs/routing_policy.md). It returns a RoutingDecision carrying selection, routing_backend, routing_reason, sidecar_used, allow_rg_fallback. Priority order (routing_policy.md §"Unified tg search decision tree"):
--index → TrigramIndex (highest override)
--gpu-device-ids → NativeGpuBackend (overrides warm-index + size routing; must fail loud if unhonorable)
--force-cpu/--cpu with structured output or no usable rg → NativeCpuBackend
- AST command →
AstBackend
- Warm non-stale compatible
.tg_index → TrigramIndex
- corpus > calibrated threshold and GPU available and calibration positive →
NativeGpuBackend
- else, plain-text request ADMITTED by
native_can_serve_plain_text → NativeCpuBackend (routing_reason = plain-text-native; skips the rg spawn entirely — see the clause bullet below)
- else,
rg available and no structured output → RipgrepBackend
- else →
NativeCpuBackend
- native CPU route fails and
allow_rg_fallback → RipgrepBackend final fallback
Load-bearing consequences:
rg is the normal cold-path backend when installed. Native CPU is the default only for structured output (--json/--ndjson), explicit --cpu, warm index, AST, and GPU fallback. Do not "optimize" tg to beat rg on cold text — that is the parity tier (see Known-Weak §3).
- Warm-index auto-routing is gated: pattern ≥ 3 bytes, no
-v, -C, --max-count, -w, -g, and the cache must exist + be non-stale + index-compatible (routing_policy.md notes). JSON/NDJSON no longer bypass a warm index.
- The plain-text native admission is a fail-closed SUBSET, not a flag (perf: skip the
rg subprocess). native_can_serve_plain_text (grep -n "pub const fn native_can_serve_plain_text" rust_core/src/routing.rs — :354 as of this pass) is the single predicate deciding when the in-process native CPU engine may answer a plain-text search instead of spawning rg. Every clause is a refusal: cheap refusals first (plain_text_native_cheap_checks_pass — :369: only PLAIN_TEXT_NATIVE_ALLOWED_FLAGS flags, no structured output, no explicit --format, stdout not a terminal, $RIPGREP_CONFIG_PATH not set-and-non-empty, PATH explicit, exactly one non-empty pattern, exactly one path, that path a regular file and not the - stdin sentinel), then the two expensive clauses evaluated last as a latency contract (pattern is native-renderable; the single path renders identically — a full-content probe: no \r, valid UTF-8, no NUL, ≤ 512 KiB, reported size == bytes read). The admitted route is RoutingDecision::native_cpu_plain_text() (:488) → NativeCpuBackend / routing_reason = plain-text-native with allow_rg_fallback = true, so a native failure still falls back to real rg; in route_search the predicate sits directly in front of the rg arm (!config.native_plain_text is the only thing standing between an admitted request and the rg subprocess — :603-613). Anything outside the subset keeps spawning rg, unchanged. Full clause list + rationale: §"Admitted plain-text native subset" ( as of this pass).
The registration sites (miss one → silent misroute)
This is a universal bug class: "register in N places, miss one, fail quietly." The CI registration-completeness gate has been BLOCKING since v1.17.1 / #282 (AGENTS.md, grep -n "registration-completeness gate is BLOCKING" AGENTS.md — :889 as of this pass; the pattern this line used to carry, BLOCKING since v1.17.1, matches NOTHING — the actual sentence is "As of v1.17.1 (#282), the CI registration-completeness gate is BLOCKING", so grep a substring of THAT), but you still author all sites by hand.
Since #977, PR CI is no longer a full routing/parity oracle for docs-only PRs. A cheap changes job (.github/workflows/ci.yml:36) detects whether the PR diff touches code (src/, rust_core/, tests/, .github/workflows/, pyproject.toml, Cargo.toml, Cargo.lock, uv.lock), and the expensive/cross-platform jobs carry needs: [smoke, changes] with if: github.event_name != 'pull_request' || needs.changes.outputs.code == 'true' (10 jobs as of this pass — re-grep needs: \[smoke, changes\]). A skipped job counts as SUCCESS for branch protection, so a docs-only PR's green rollup proves nothing about code behavior; main pushes always run the full matrix (the job forces CODE_FILES="main-push" off-PR). Do not cite a docs-only PR's CI as routing evidence, and do not propose paths-ignore on required checks (branch protection would wait forever on a run that never starts).
A new top-level tg COMMAND needs four sites (AGENTS.md "Adding a Command or Flag", starting at line 396; re-derive by grepping the header, not the line number, since AGENTS.md's line numbers shift as sections are added above it):
| # | Site | File |
|---|
| 1 | KNOWN_COMMANDS set | src/tensor_grep/cli/commands.py:9 |
| 2 | Commands::X variant + dispatch arm | rust_core/src/main.rs:910 (enum, unchanged this pass); e.g. Commands::Prepare/Commands::Ledger dispatch arms — grep -n "Commands::Prepare|Commands::Ledger" rust_core/src/main.rs (was :6691/:6686, now :6966/:6961) |
| 3 | PUBLIC_TOP_LEVEL_COMMANDS (parity test) | tests/e2e/test_routing_parity.py:46 |
| 4 | @app.command function | src/tensor_grep/cli/main.py |
A new search flag needs two front doors (AGENTS.md, same section, "two front doors") or it leaks to ripgrep and crashes with rg: unrecognized flag for anyone on the published binary:
| # | Site | File |
|---|
| 1 | SEARCH_PYTHON_PASSTHROUGH_FLAGS (native allowlist) | rust_core/src/main.rs:204 |
| 2 | bootstrap._TG_ONLY_SEARCH_FLAGS (Python front-door allowlist) | src/tensor_grep/cli/bootstrap.py:50 |
A new MCP tool (or a request/response shape change to an existing one) is a FIFTH registration site — distinct from the four command sites above (AGENTS.md "Adding a Command or Flag", 5th-registration-site note). Every MCP tool's JSON envelope embeds mcp_contract_version from the SINGLE constant _TG_MCP_SERVER_CONTRACT_VERSION (grep -n "_TG_MCP_SERVER_CONTRACT_VERSION = " src/tensor_grep/cli/mcp_server.py — :138 as of this pass, value "1.7.0"); _inject_mcp_contract_fields (grep -n "^def _inject_mcp_contract_fields" src/tensor_grep/cli/mcp_server.py — :1125) HARD-assigns it into every serialized tool envelope (a stale per-tool literal can never win — the M14 retirement of setdefault), and the same constant sets server._mcp_server.version (:185). Bump the constant whenever any tool's request/response shape changes — the tg_find MCP PR (#627) shipped with an un-bumped contract version and only the mandatory adversarial Opus gate caught it, not tests or CI.
Blind spot to internalize: tg callers <fn> finds callable registration sites in ~1s, but the call graph cannot see set/list/decorator registrations — _TG_ONLY_SEARCH_FLAGS is a set, @app.command is a decorator, the Rust dispatch is a match arm. Those are the sites most often missed (--rank lived in a set). So tg callers for the reachable ones and grep / tg scan for the declarative ones, then confirm your entry appears in all sites. (The actual add-a-thing procedure lives in tensor-grep-change-control; this skill only explains why the sites exist.)
Unknown and reserved top-level commands fail closed on BOTH front doors (A90). commands.py carries RESERVED_TOP_LEVEL_COMMANDS (grep -n "RESERVED_TOP_LEVEL_COMMANDS = " src/tensor_grep/cli/commands.py — :77 as of this pass, with the A90 lifecycle comment :67-76): roadmap command names that DO NOT EXIST yet, kept disjoint from KNOWN_COMMANDS (:9) by a test-pinned RESERVED ∩ KNOWN == ∅ invariant — realizing a reserved name means removing it from the reserved set in the same change. A genuinely unknown command is never forwarded to search; both doors refuse it with exit 2, diagnostic on stderr, stdout EMPTY, and a did-you-mean suggestion: the Python bootstrap via _emit_unknown_command_human / _emit_unknown_command_json (grep -n "_emit_unknown_command" src/tensor_grep/cli/bootstrap.py — :444/:454 as of this pass, then raise SystemExit(2) at :1595) and the native binary via top_level_unknown_command_refusal (grep -n "top_level_unknown_command_refusal" rust_core/src/main.rs — refusal block :1347-1373 as of this pass, std::process::exit(2); human text when --help/-h is present, otherwise a single {"error": {"code": "unknown_command", ...}} JSON object on stderr).
Backend Fail-Closed Contract
The single most important correctness invariant. src/tensor_grep/backends/base.py defines it: every ComputeBackend MUST raise BackendExecutionError on a real failure — never return a clean empty / 0-match SearchResult, and never silently swap to an engine that cannot preserve the requested semantics.
Why a context tool cannot afford to violate it: a swallowed backend failure reaches a coding agent as a trustworthy "no matches." That is the one lie a search tool must never tell — the agent then edits on the belief that the symbol does not exist.
Rules when a path can fall back (AGENTS.md "Backend Fail-Closed Contract", backends/base.py:7):
- Fail closed for any flag/contract the fallback cannot preserve.
--pcre2 through a non-PCRE2 engine ⇒ raise, do not swap (that produces wrong results, not just slower ones).
- A legitimate degraded fallback must be VISIBLE: set
fallback_reason (and a distinct routing_reason) on the result so JSON/CLI consumers can tell degraded output from real output. Never label heuristic output as model output.
- Validate an untrusted response shape before indexing (e.g. a model's class count vs a fixed label list) so a mismatch degrades gracefully instead of raising an
IndexError a broad except then swallows.
The recurring anti-pattern: a bare except Exception: that returns empty or falls through to a different engine. This has been fixed repeatedly across audits — the Rust/PCRE2 bridge, the ast-grep OOM mask, the tree-sitter query swallow, CyBERT classify. When you review/write any backend or router that can change engines, this is the first thing to check. The structural fix (a SafeBackendMixin + a fault-injection conformance CI gate) is planned but not yet shipped, so the discipline is still per-file. The same rule extends to routers: an explicit --gpu request silently routed to CPU must raise/emit a diagnostic, not swap silently.
A new command does not inherit a sibling's fail-closed boundary-catch automatically — prove it, don't assume it (tg find, v1.77.0, #189). tg find and tg search --semantic share the same dense-embedding core (retrieval_dense.py/retrieval_fusion.py), but their fail-closed SHAPE differs because their corpora differ: --semantic re-ranks an already regex-prefiltered match set, so a degrade to BM25-only is always cheap and benign; tg find walks and ranks the WHOLE repo with no prefilter, so a query-time model fault reachable mid-walk is a materially different risk surface. The first tg find build wave shipped WITHOUT a command-boundary catch for DenseUnavailableError — it would have propagated as an uncaught crash instead of a visible BM25-degrade — caught only by the mandatory adversarial Opus gate, not by the (green) unit tests, and fixed in the same PR (045fadc). Rule: when a new command reuses an existing backend/compute path, verify its OWN command-boundary exception handling explicitly; do not assume "the underlying module already has a fail-closed contract" is sufficient — the CALLER must also catch and degrade/exit correctly at ITS boundary. See tensor-grep-run-and-operate §11c for tg find's full exit-code contract (BackendExecutionError→exit-2; empty+result_incomplete→exit-2 else exit-1; found+result_incomplete→print then exit-2).
Partial-results contract: suppression != absence (SearchResult.result_incomplete)
Companion invariant to the Backend Fail-Closed Contract above, shipped in round-4 slice 3 (#341, commit f11ce28, v1.18.x). SearchResult (grep -n "^class SearchResult" src/tensor_grep/core/result.py — :71, unchanged this pass; fields — grep -n "result_incomplete: bool\|incomplete_reason: str" src/tensor_grep/core/result.py, was :54-55, now :111-112) carries result_incomplete: bool = False and incomplete_reason: str | None = None, deliberately not overloaded onto fallback_reason — fallback_reason means "the execution engine was swapped"; result_incomplete means "this engine ran, but a soft per-item error suppressed part of the output." Conflating them would emit a false "we fell back" signal to doctor/JSON consumers.
The trigger: rg exit code 2 is a soft per-file error (e.g. one unreadable/missing path among many) and rg still emits matches for every readable file. Before #341, tg's parser raised unconditionally on exit > 1, discarding those partial matches — and even if it hadn't, tg would have silently exited 0 while rg exits 2 (a parity break an agent scripting around exit codes would never see).
And the exit-code side of this contract has since been made STRICTER, not looser — do not describe it as "empty partial -> exit 2, non-empty partial -> exit 0." #398 first made ANY truncated partial exit 2; #399 briefly walked that back to exit-2-only-when-empty; #401 reverted #399 after a unanimous design council — the current, final contract is: any result_incomplete/partial result exits 2 regardless of whether matches were found, because a truncated match/caller/blast-radius list must never be silently trusted as exhaustive. See tensor-grep-large-repo-scale-campaign §5 for the full exit-code table and docs/CONTRACTS.md — grep -n "This mirrors \tg search`'s `2 = result_incomplete`"(was:114, now :157`; CONTRACTS.md grows fast, re-grep before citing) for the symbol-command three-state exit-code contract.
The 5-site fix, cite file:line:
- Parse-first-then-branch —
backends/ripgrep_backend.py (search, _search_files_with_matches, _search_counts): exit 2 with a non-empty parse keeps the results, sets result_incomplete=True + a stderr-derived incomplete_reason (grep -n "result_incomplete = True" src/tensor_grep/backends/ripgrep_backend.py — :144,325,444, essentially unchanged this pass); exit >2, or exit 2 with nothing parsed, raises BackendExecutionError (RESOLVED #79/#10/#14, commit a7c9431: every RipgrepBackend fatal path, including the rg-missing guard — grep -n "requires the 'rg' binary" src/tensor_grep/backends/ripgrep_backend.py (was :505, now :541) — now raises BackendExecutionError instead of a bare RuntimeError, so cli/main.py's per-file except BackendExecutionError CPU-fallback retry — grep -n "except BackendExecutionError" src/tensor_grep/cli/main.py (was :8005, several call sites exist today, e.g. :4796/:8280/:8396; re-grep and confirm which one is the per-file retry before citing a single line) — catches it instead of falling into the broad except Exception and crashing the whole search — see code-search-and-retrieval-reference §1 for the exit-code table).
- Monotonic merge —
merge_runtime_routing (grep -n "^def merge_runtime_routing" src/tensor_grep/core/result.py — was :135, now :142) OR-merges result_incomplete across sub-results (aggregate.result_incomplete or result.result_incomplete), so the CLI/MCP/sidecar aggregate inherits uniformly — any incomplete sub-result taints the whole.
- Exit-code parity —
grep -n "if .*result_incomplete else\|exit_incomplete else" src/tensor_grep/cli/main.py (was a contiguous :8175-8240 block, now scattered across :8549-8620; the wiring moved and split, do not assume a contiguous range): the terminal exits read sys.exit(2 if … result_incomplete/exit_incomplete else …) across the files-with/without-matches, , quiet, and post-format branches, closing the "tg exits 0 while rg exits 2" gap.
Rule for any new path that can drop some results due to a soft/partial failure: set result_incomplete + incomplete_reason. Do not (a) raise and lose the good results, or (b) silently return only the good results as if they were the complete answer — that is the same "suppression reads as absence" lie the Backend Fail-Closed Contract forbids, just at the partial-result layer instead of the total-failure layer. Tests: tests/unit/test_rg_exit2_partial.py.
Native-delegation forward-or-refuse contract (_can_delegate_to_native_tg_search)
_can_delegate_to_native_tg_search — grep -n "^def _can_delegate_to_native_tg_search" src/tensor_grep/cli/main.py (was :3709, then :3794, now :4095 as of 2026-08-14) — gates whether a Python-side tg search hands the entire search to the native tg subprocess (_build_native_tg_search_command — grep -n "^def _build_native_tg_search_command" src/tensor_grep/cli/main.py, was :3731, then :3816, now :4117 as of 2026-08-14) and then sys.exit()s on its result — a delegation that runs before the Python-side BM25 rerank (--rank) and the in-backend sort (--sort-files) ever execute.
The invariant: delegation is permitted only when native execution is byte-equivalent to the Python path for the requested config. The gate enforces this mechanically, not by convention — it loops every field name in _NATIVE_TG_DELEGATION_DEFAULT_REQUIRED_FIELDS (grep -n "_NATIVE_TG_DELEGATION_DEFAULT_REQUIRED_FIELDS = " src/tensor_grep/cli/main.py -- :1966 as of 2026-08-14, was :1894 onward) and refuses delegation (falls through to the Python/backend path) if any of those fields differs from a fresh SearchConfig()'s default. Every SearchConfig field must land in exactly one bucket:
- Forwarded — read by
_build_native_tg_search_command and translated into native argv.
- Refused — listed in
_NATIVE_TG_DELEGATION_DEFAULT_REQUIRED_FIELDS, so a non-default value forces the gate closed.
- Gate-handled — read off explicit keyword args at the call site (
files_with_matches, files_without_match), not the config object.
- KNOWN_GAP — explicitly documented pre-existing tech debt, tracked rather than silently dropped.
This is enforced by a governance ratchet, tests/unit/test_native_delegation_field_coverage.py (round-4 #25, shipped as #342, commit 5e6f780): it AST-derives the "forwarded" set straight from _build_native_tg_search_command's source (ast.walk over every config.<attr> read), so that list can never silently drift from the real code, then asserts all_fields - (forwarded | required | gate_handled | known_gap) == set(). Add a new SearchConfig field and forget to classify it → this test goes red immediately.
The bug this closes (#342): rank_bm25 and sort_files were neither forwarded to native argv nor in the refuse-tuple, so tg search --rank --cpu silently delegated to the native binary — which has no BM25 of its own — and sys.exit()d before the Python rerank/sort ever ran, returning unranked/unsorted output that looked like a normal, correct result (suppression indistinguishable from absence, same class the partial-results contract above targets). This is the same flag-drop bug class as the -u/-uu no-op fixed in #336 (round-4 PR-A slice 1): a flag parses successfully but never reaches the engine that must honor it.
Landmine already hit once — do not re-propose it: the tempting "just gate on any field differing from defaults" fix is wrong. query_pattern is auto-set to the search pattern on every search, so a differs-from-default check would always see a difference and refuse delegation on every call, killing the fast path entirely (the exact failure mode from the 2026-06-30 #1 audit finding — see tensor-grep-failure-archaeology). The fix has to be per-field, not "any field changed."
Rule when adding a new SearchConfig field that affects search output: decide immediately whether native delegation can reproduce it byte-for-byte. If not, add the field name to _NATIVE_TG_DELEGATION_DEFAULT_REQUIRED_FIELDS. The ratchet test refuses to let you skip this decision silently — it is a hard gate, not a lint suggestion.
A THIRD rg-passthrough door lives INSIDE cli/main.py::search_command — gated on rg availability, not a platform flag (task #24, 2026-07-30)
This is a third, independent rg-passthrough decision, distinct from both the bootstrap front door (bootstrap._run_rg_passthrough, "The front door" above) and the Rust routing.rs tree ("Native-vs-Python routing" above). It lives entirely inside the Python Typer app, fires only for invocations that a _TG_ONLY_SEARCH_FLAGS flag has already forced past the bootstrap front door (e.g. --stats, --ast, --rank, --semantic), and is easy to miss because nothing about it is platform-conditional — yet it produced a real Windows-vs-Linux CI divergence (docs/BACKLOG.md "STILL OPEN — tg search --stats routing DIVERGES BY PLATFORM").
The walk-ceiling fast-refuse: 3 doors, 2 constants, 1 value (A9, v1.92.3/#702)
Before #702, the plain flag-less bootstrap._run_rg_passthrough path (grep -n "^def _run_rg_passthrough" src/tensor_grep/cli/bootstrap.py — was :1088, now :1421 — the front
door a bare tg search PATTERN with no scoping flags hits, before main.py's Typer app is ever
reached) had no walk ceiling at all. main.py's three vendored/workspace/large-root refusal guards
never ran for this path, so an unscoped search on a large defaulted-path root silently walked unbounded
until it hit the 60s TG_RG_TIMEOUT_SECONDS subprocess backstop — natively reproduced, not a WSL
filesystem artifact.
The fix is one constant, enforced coherently across 3 doors, not three independent numbers that can
drift apart:
- The single constant:
IMPLICIT_SEARCH_WALK_FILE_CEILING = 1500
(src/tensor_grep/io/scan_limits.py:106 — moved here from io/directory_scanner.py since the
v1.93.2 pass; directory_scanner.py:34 now only re-imports/re-exports it, so a grep of the old
file still finds a hit but not the definition).
- Door 1 — Python bootstrap probe:
bootstrap._search_paths_include_oversized_implicit_root
(grep -n "^def _search_paths_include_oversized_implicit_root" src/tensor_grep/cli/bootstrap.py — was :804, now :1053), gated on paths_defaulted (fires only when no explicit PATH was given, not on
every search).
- Door 2 — Python Typer app:
main.py's _LARGE_ROOT_SCAN_FILE_CEILING = IMPLICIT_SEARCH_WALK_FILE_CEILING
(grep -n "_LARGE_ROOT_SCAN_FILE_CEILING = IMPLICIT_SEARCH_WALK_FILE_CEILING" src/tensor_grep/cli/main.py — was :5150, now :5363), the alias that keeps the Typer-app-side ceiling from silently drifting from the
bootstrap door's value.
- Door 3 — Rust native front door:
rust_core/src/rg_passthrough.rs keeps its own copy of the same
numeral (pub const IMPLICIT_SEARCH_WALK_FILE_CEILING: usize = 1500;, rg_passthrough.rs:153),
synced by convention (not a shared cross-language build constant) — a future change to the
Python-side value needs a matching edit here or the two front doors will disagree on where the
ceiling sits.
Escape hatches: an explicit PATH, --max-depth, or --allow-broad-generated-scan — --glob/
--type alone do not bypass the ceiling when the path itself was defaulted. Result: an over-ceiling
implicit root now refuses in ~1.7s (exit 2) instead of silently walking for up to 60s.
The dynamic_unresolved honesty marker — every downstream consumer must re-check it, not inherit it (A10/A15, v1.93.0/#703 + v1.93.2/#709)
tg imports/tg importers/tg blast-radius mark a relative dynamic import
(import_module(".x", package=...), __import__(..., level>=1)) as dynamic_unresolved rather than
resolving it to a guessed target — the literal text is preserved in unresolved, and it is never
silently pointed at a same-named decoy top-level file (both the forward tg imports direction and the
reverse tg importers direction). Absolute-literal dynamic imports (import_module("pkg.mod")) still
resolve normally ("dynamic": true) — only the genuinely ambiguous relative/computed form degrades to
the honesty marker. Rule: a wrong edge is worse than a missing one.
The #709 lesson is the reason this gets its own subsection instead of living as a one-line note next
to #703: shipping the honesty marker at the import-graph layer (#703) was NOT sufficient by itself —
tg blast-radius's reverse scoring prefilter had its own, separate code path that fuzzy-matched
dynamic_unresolved literals against real symbol names, so a same-named decoy could still leak into
affected_files/dependent_files through the scoring layer even though the import-resolution layer
correctly refused to link it. #709 fixed the prefilter to exclude dynamic_unresolved literals too,
with a pinned ranking test proving zero legitimate reorder. Generalize this: when a marker like
dynamic_unresolved is introduced at one layer (import resolution), audit every OTHER layer that reads
import/symbol data for its own independent path that could re-introduce the same class of false edge
(a scoring prefilter, a cache, a graph-traversal shortcut) — do not assume a single fix point closes the
whole surface.
Cross-domain native-binary detection (A11, v1.93.0/#704)
is_cross_domain_native_binary (runtime_paths.py:472) decides whether a resolved tg binary lives in
a different OS/filesystem domain than the current process (the concrete case: a WSL Linux process
resolving a Windows-built tg.exe via a translated /mnt/c/... path). Before #704, cross-domain
detection was .exe-suffix-only — but the managed installer also ships a bare-named POSIX shim
tg that wraps tg.exe, and that shim has no .exe suffix to detect. The bare shim was misclassified
as same-domain, so its sentinel probe used an untranslated /tmp/... path against the Windows
binary and failed with a confusing path_not_found/failed_probe_path — a probe bug, not a genuine
GPU unavailability signal. The fix adds two more signals: a sibling tg-native-metadata.json file, and
a co-located <name>.exe file next to the bare-named shim — both checks are fail-closed-only
(reading the metadata file is capped at 1MiB and guarded against OSError/ValueError; a read failure
never promotes a binary to cross-domain, it only affects whether the extra signal is available) and
non-WSL hosts never run these checks at all, so the fix cannot introduce a false-positive on a
plain Windows or Linux box. Post-fix, the same WSL bare-shim probe reports an honest
status=unsupported, routing_backend=NativeCpuBackend, routing_reason=gpu-auto-fallback-cpu, exit 0
instead of the misleading path error.
MatchLine is a frozen, HASHABLE dataclass
grep -n "^class MatchLine" src/tensor_grep/core/result.py (was :4-17, now :55; :4 is strip_line_terminator, a different symbol entirely — always grep the class name, never trust the old range) (@dataclass(frozen=True) class MatchLine). submatches (tuple[dict[str, object], ...] | None, added by #340 to carry rg's per-occurrence byte offsets for --vimgrep/--column) is a tuple-of-dicts — dicts are unhashable, so a populated submatches would break hash(MatchLine(...)) the moment a frozen dataclass's default hash implementation (derived from its ==-participating fields) tried to hash it.
The fix (#344, commit 80de0b4): submatches: tuple[dict[str, object], ...] | None = field(default=None, compare=False). compare=False excludes the field from both __eq__ and the derived __hash__, so MatchLine stays hashable even when submatches is populated. Excluding it from == is intentionally correct, not a shortcut: the offsets are a pure function of text + line_number, so two matches equal on those fields are equal regardless of any incidental difference in their submatch tuples.
No caller hashes MatchLine today — this was caught as a latent landmine before any set/dedup consumer existed, not a live crash. Treat it as the standing precedent: this codebase keeps its frozen dataclasses hashable on purpose. Any new field added to a frozen dataclass here that is itself unhashable (a list, dict, or other mutable/unhashable container) must be marked field(..., compare=False) — or, if it genuinely must participate in equality, the dataclass needs a deliberate eq=False/custom __hash__ redesign, not a silent break.
Adjacent, same commit: the per-match submatch stash is now built only behind config.vimgrep or config.column in RipgrepBackend.search — only those two formatters consume the offsets, so building the tuple on every default-format match was wasted allocation (found via the blast-radius-regression profile that produced #345, not a separate bug hunt). Output stays byte-identical; --vimgrep/--column still emit one row per rg occurrence.
ASCII-only CLI output contract
tg does not reconfigure stdout to UTF-8, and Windows consoles commonly default to the cp1252 codepage. typer.echo (used throughout the CLI) raises UnicodeEncodeError on any character outside that codepage — a hard crash, not mojibake. #346 (commit 6b7b518) found render_inventory_text in src/tensor_grep/cli/inventory.py emitting a literal ⚠ (U+26A0 WARNING SIGN) on the truncation-notice path (repo > max_files); on a stock Windows terminal, tg inventory on a large repo crashed instead of printing a warning.
Rule: no non-ASCII characters in any tg-CLI-rendered text output (Typer echo/print call sites — this governs strings tg itself prints, not file contents being searched). Use bracketed ASCII markers instead — the fix replaced ⚠ with the literal string [!]. Before adding a new CLI-rendered string (a warning glyph, a checkmark, box-drawing table characters, an arrow), check it is str.isascii()-clean; if in doubt, dogfood on a real cp1252 Windows console, not just a UTF-8-default terminal or CI runner — CI's UTF-8 locale will not catch this class of bug, it is Windows-console-only and was found by dogfooding a large real repo locally (tensor-grep-dogfood-real-corpus-before-shipping-precision-2026-07-03 memory), not by the fixture test suite.
The moat: agent-native context, not faster grep
Positioning is a design constraint, not marketing. tg is not a faster grep. ripgrep is the raw-text parity baseline; ast-grep is the structural-search baseline. The moat is the agent-native code-intelligence layer: orient, callers, blast-radius, defs, refs, source, agent (the capsule), session, find (whole-repo hybrid NL search, v1.77.0, #189). Peers to know: Aider repo-map (tree-sitter + NetworkX PageRank, --map-tokens), Sourcegraph Cody (SCIP + BM25 + embeddings → rerank), Cursor (index-first embeddings + Merkle change detection).
Engineering-capacity consequence (AGENTS.md "Roadmap Sequencing 2026-07-02"): CPU-only, every-install moat work is funded first — local hybrid semantic search (BM25 + CPU dense embeddings + RRF, no API key), tg registration-check as a first-class command, a Bloom-filter n-gram chunk prefilter — before advancing the GPU program. Never make a change that implies "tg beats rg for cold exact-text search."
Invariants that must hold (agent contract)
These are enforced by the capsule/context contract (docs/CONTRACTS.md §3, "Context and edit-planning contracts") and the agent-readiness gate. A change that breaks one is a contract regression even if tests are green.
context_consistency — edit_plan_seed.primary_file, navigation_pack.primary_target.file, the rendered source sections, and follow-up read commands must not contradict each other. The payload reports whether the primary file is included, whether rendered context matches the target, whether confidence was downgraded, and why a primary file was omitted (docs/CONTRACTS.md, grep -n "reports whether the primary file is included" — was :92, now :134).
- Ambiguity hard-stop — when equal-confidence alternatives are unresolved, cap
confidence.overall and primary_target.confidence below the edit threshold, set ask_user_before_editing.required = true, and mirror it in top-level ambiguity.status = "tie_requires_confirmation". A validation-resolved tie records ambiguity.status = "tie_resolved", resolved_by = "targeted-validation" with concrete resolution_evidence; an LSP-resolved tie needs explicit provider-response proof (docs/CONTRACTS.md, grep -n "tie_requires_confirmation" — was :106, now :149). This is the safety floor added in #302 — do not weaken it.
- Validation provenance — validation hints use
validation_plan[].detection ∈ {detected, heuristic, generic} and must align with the primary target language: a TS target must not silently get pytest, a Python target must not silently get npm test; validation_alignment records filtering. JS commands require package.json evidence, Python commands require test/marker/layout evidence, and commands are omitted entirely when no runner evidence exists — never invented (docs/CONTRACTS.md, grep -n "validation_alignment.*records whether" — was :94, now :137).
- Evidence labeling — routing/claim evidence is labeled
parser-backed | rg-backed | graph-derived | heuristic | LSP-confirmed | stale/uncertain; when signals disagree, downgrade confidence and surface the contradiction rather than hiding it behind one ranked file (docs/CONTRACTS.md, grep -n "parser-backed" — was :109, now :152). LSP availability is semantic proof: a row counts as only with an explicit (, — was , now ).
Known-weak points (state plainly, never oversell)
Encode these honestly; the dogfood report itself emits world_class_readiness.status = "not_claimed" (docs/CONTRACTS.md, grep -n "world_class_readiness" — was :86, now :128). Everything unproven stays labeled candidate/experimental. Experimental, default-OFF: GPU, LSP, semantic, CyBERT/provider paths.
- Flat, no-IDF ranking scorer. Repo-map scoring uses flat integer term counts (
_score_text_terms/_score_file_path/_score_symbol in src/tensor_grep/cli/repo_map.py -- _score_text_terms:8189, _score_file_path:8377, _score_symbol:8454 as of 2026-07-27; re-derive with: grep -n 'def score' src/tensor_grep/cli/repo_map.py | sort; these drift every release), not IDF/term-rarity weighting. Ranking surfaces (search --rank, the agent capsule, semantic) can silently flip on a corpus change, and the blast radius of a ranking change is invisible to the call graph. A degrade-to-ask safety floor was added in #302; the flat scorer itself remains open debt. Treat any ranking-affecting change as high-risk and benchmark it.
- GPU Phase-0 SHIPPED (v1.75.0-v1.75.4, PRs #593-#597) but no speed crossover is proven, and the shipped kernel is NOT what the roadmap language implies. NVIDIA native assets are built and locally correctness-proven (RTX 4070
sm_89 / RTX 5070 sm_120, 1GB/5GB match+file-set correctness -- docs/gpu_crossover.md), but gated OFF the public release by the CI Actions var TENSOR_GREP_RELEASE_NATIVE_ASSET_PROFILE (default native-frontdoor, CPU-only; GPU asset publishing needs the non-default native-frontdoor-gpu). Phase 1 (publishing those already-built assets) is now a reversible flag-flip, not a multi-week rebuild -- but flipping the var publishes assets only: it does not promote GPU, does not change the CPU-default auto-recommendation, and does not prove a speed crossover. The shipped kernel (gpu_text_search_positions) is a position-parallel brute-force byte-compare, NOT a PFAC/Aho-Corasick automaton (docs/gpu_crossover.md:133-138 — PFAC remains documented future work, not what runs today). No crossover is proven at ANY scale, including the best-case many-fixed-pattern lane (100 patterns over 1GB: fair-baseline rg -F -e ... -e ... at 0.169s vs the GPU-requested path at 0.448s, itself a CPU-fallback measurement, not even a real GPU number); historical worst case at 5GB is ~30-35x slower than rg. Keep the honesty floor verbatim: no speed crossover is proven vs rg/tg_cpu, GPU auto-recommendation stays false, and the reviewer-gated speed-crossover gate remains unmet (, — was , now ). (CEO decision package, task-store #169 — not a GitHub issue, re-verify with ); release checksums currently ship 3 CPU-only rows. Explicit stays supported and must fail loud when unhonorable; sidecar-routed GPU output is compatibility evidence, never GPU-acceleration proof.
Domain background a mid-level engineer may lack
- ripgrep internals: default regex engine matches invalid UTF-8; PCRE2 requires valid UTF-8 and transcodes (hence
--pcre2 cannot be silently swapped); binary detection via NUL byte; exit 2 = non-fatal error, exit 1 = clean no-match, exit 0 = match; -- ends options; -e supplies patterns; -uuu = --no-ignore --hidden --binary.
- BM25 / IDF: IDF weights rare terms higher; the flat scorer above skips this, which is why a common term can dominate ranking after a corpus grows.
- PyO3 + the GIL: release the GIL (
py.detach/allow_threads) for CPU-bound or subprocess work; mock-based FFI tests can pass while the real extension is dead — verify FFI against the real built extension (maturin develop), not mocks.
- MCP argv surface: MCP tool handlers forward LLM-controlled params into
tg/rg/git argv — a flag-injection surface. List-argv (shell=False) stops shell injection but not flag injection; a -- sentinel before user positionals is required. Security detail lives in tensor-grep-failure-archaeology and the round-3 hardening notes in AGENTS.md.
Fast self-check before you trust a claim about this design
# Front door + version identity
uv run tg --version # expect: tensor-grep 1.110.14 (or current)
# The published entry point (must be bootstrap.main_entry, not a Typer callback)
uv run python -c "import tensor_grep.cli.bootstrap as b; print(b.main_entry)"
# Routing / launcher observability
uv run tg doctor --json | python -c "import sys,json;d=json.load(sys.stdin);print(d.get('search_acceleration_backend'), d.get('path_tg_first_launcher_kind'))"
# Fast readiness gate (context_consistency, parity edges, registration, capsule invariants)
uv run python scripts/agent_readiness.py --output artifacts/agent_readiness.json
Never claim a speedup, a fixed weak point, or "tests pass" from a model self-report. Confirm against external state: an exit code, a real-binary dogfood, a file:line that resolves. A subagent's "green" is a hypothesis until then.
Provenance and maintenance
All facts verified against the live repo on 2026-07-08 at v1.49.3; the tg find fail-closed-boundary
paragraph and moat-command-list addition were verified 2026-07-16 at v1.78.1; a consolidated grep pass
on 2026-07-22 at v1.93.2 re-verified and corrected all 7 previously-drifted file:line cites
(front-door entry point, command registration sites, flag front doors, and the 3 native-delegation
cites) and added the A9/A10-A15/A11 subsections plus the A3 backend_cpu.rs-vs-native_search.rs
split — the rest of the file (Invariants, moat, ASCII-output, MatchLine sections) was not re-walked
line-by-line in that pass. A further consolidated re-verification pass on 2026-07-24 at v1.95.0
walked every file:line cite in this file against origin/main — two minor releases carrying the
Java/C#/PHP symbol-graph language campaign had grown cli/main.py, repo_map.py, and (for reasons
unrelated to that campaign) bootstrap.py enough to drift most numeric cites, several by 1000+ lines
— and corrected: the front-door entry point + forwarding range + 4 more bootstrap.py cites; both
native-delegation gate/builder/refuse-tuple line numbers; the flag-front-door _TG_ONLY_SEARCH_FLAGS
line; the exit-code-wiring and per-file-CPU-fallback-retry ranges in cli/main.py; the 3 flat-scorer
line numbers in repo_map.py (which alone moved ~1700-1800 lines); the mcp_server.py MCP-envelope
sites; runtime_paths.py's cross-domain-detection line; the rg_passthrough.rs sentinel/test line
numbers; the subprocess_policy.py rg-timeout line; several docs/CONTRACTS.md and AGENTS.md cites;
and the IMPLICIT_SEARCH_WALK_FILE_CEILING constant's module, which moved from
io/directory_scanner.py to io/scan_limits.py (a symbol-moved case, not just a line drift — the old
file now only re-exports it, so a naive grep there still finds a hit and can mask the move). This pass
also made one semantic, not merely numeric, correction: the AST-routing bullet under "Native-vs-Python
routing" claimed AstBackend.is_available() requires a CUDA device — that GPU/torch_geometric gate
was deleted as dead code back in v1.64.4/#542 (is_available() has checked only tree_sitter
presence since), a drift this skill carried unnoticed across ~30 releases and the entire v1.93.2 pass.
The practical routing OUTCOME ( still favors the CLI sidecar on a typical box) is
unchanged, but it is now correctly attributed to a DSL-consistency policy choice
( — , was , now -- cite by symbol; the older range now holds unrelated #299 code), not a hardware gate. — (->), (->), and (->) had all drifted; only () held. The ripgrep_backend.py -setting lines (§ Partial-results contract) stayed close ( vs the cited ), but its rg-missing guard drifted (->).
2026-08-13 pass (v1.110.14 tree, 568065a) — six corrections, every one verified against this tree before writing:
(1) the --quiet rg-passthrough blast radius (third-door section, last bullet) was still described as OPEN with "zero mentions of quiet in ripgrep_backend.py" — FALSE in this tree: search_passthrough appends -q (ripgrep_backend.py:511) with the four-consumers/one-streams hazard comment (:492-510) and tests/unit/test_quiet_survives_rg_passthrough.py pins placement; rewritten CLOSED.
(2) _select_ast_backend_for_pattern MOVED: main.py's copy (old pins :6737/:6915) is now a forwarding shim (:7270); the real implementation + wrapper-preference block is in cli/ast_workflows.py (:1183, preference call :1241); ast_wrapper.is_available() has zero occurrences in main.py, so the old grep instruction was dead — both cites re-pointed.
(3) Added the 5th registration site: the MCP tool contract version _TG_MCP_SERVER_CONTRACT_VERSION (mcp_server.py:138, hard-injected into every envelope by _inject_mcp_contract_fields :1125).
(4) Routing taxonomy gained the native_can_serve_plain_text admission clause (routing.rs:354/:369/:488, route_search rg-arm gate :603-613; docs/routing_policy.md:58-75).
(5) Folded #977 (the ci.yml changes job :36 + 10× needs: [smoke, changes] gating — docs-only PR CI skips the expensive jobs, main pushes always run full) and A90 (unknown/reserved top-level commands fail closed on both doors, exit 2, did-you-mean — commands.py:77, bootstrap.py:444/454/1595, rust_core/src/main.rs:1347-1373).
(6) The grep "BLOCKING since v1.17.1" in TWO places returned zero hits — the actual AGENTS.md sentence (:889) is "As of v1.17.1 (#282), the CI registration-completeness gate is BLOCKING"; both replaced with the working pattern registration-completeness gate is BLOCKING.
Re-verify anything volatile before relying on it:
- Version:
grep '^version' pyproject.toml (was 1.95.0).
- Front-door entry point:
grep -n "^def main_entry" src/tensor_grep/cli/bootstrap.py (this line has now carried THREE different stale numbers across passes -- :1398 as of 2026-07-27, :1444 elsewhere in this very file, actual today :1550 -- never trust any of them, always re-grep) and confirm pyproject.toml/packaging still points tg at tensor_grep.cli.bootstrap:main_entry.
- Command registration sites (4):
commands.py:9 (KNOWN_COMMANDS, unchanged), rust_core/src/main.rs:910 (enum Commands, unchanged; Commands::Prepare/Commands::Ledger dispatch arms -- grep -n "Commands::Prepare\|Commands::Ledger" rust_core/src/main.rs, was :6691/:6686, now :6966/:6961), tests/e2e/test_routing_parity.py:46 (PUBLIC_TOP_LEVEL_COMMANDS, unchanged), @app.command in src/tensor_grep/cli/main.py. Re-grep these, do not trust the numbers. The v1.95.0 pass called all 4 "byte-stable"; by 2026-07-29 the enum had moved 889->910, the dispatch arms 5456/5451->6691/6686 (over 1,200 lines), and PUBLIC_TOP_LEVEL_COMMANDS 18->46; by 2026-08-01 the dispatch arms had moved AGAIN, 6691/6686->6966/6961 -- a second drift on the same anchor in one release. A "confirmed stable" claim about a growing file is a claim with a short half-life, twice over now.
- Flag front doors (2):
rust_core/src/main.rs:204 (SEARCH_PYTHON_PASSTHROUGH_FLAGS, unchanged), bootstrap.py:50 (_TG_ONLY_SEARCH_FLAGS, moved from :38, unchanged since).
- Native-delegation cites (3):
grep -n "^def _can_delegate_to_native_tg_search\|^def _build_native_tg_search_command\|_NATIVE_TG_DELEGATION_DEFAULT_REQUIRED_FIELDS = " src/tensor_grep/cli/main.py -- this exact bullet previously cited :3709 for BOTH _can_delegate_to_native_tg_search AND _build_native_tg_search_command (one line number for two different functions, itself a symptom of hand-copied numbers rather than a fresh grep); actual today: , , (this last one was already correct).
If a re-verify disagrees with this skill, fix the skill — a wrong runbook is worse than none — and route any code change through tensor-grep-change-control.