| name | tensor-grep-hermetic-hostile-tests |
| description | Use when writing a gated test that must pass identically on the dev desktop AND the CI pytest env (which lacks optional engines), constructing a HOSTILE fixture (permission denial, junction/symlink swap, network partition, disk-full, killed process, corrupted file, missing binary) and proving it actually BITES before the probe runs, forcing a deterministic injectable seam instead of env-detecting, mutation-asserting that a red-arm control really applied, or adding a positive control so a zero cannot read as "clean". Triggers: "hermetic test", "hostile fixture", "fixture-BITES precheck", "STILL VACUOUS", "env-independent", "setup-not-assertion", "the fixture never applied", "mutation asserted-applied", "positive control", "Event-gated parent swap", "RED reason class", "platform arms". Sibling of tensor-grep-codex-gated-audit-loop (the loop) and tensor-grep-validation-and-qa (what counts as proof); this one is the CONSTRUCTION discipline for hermetic + hostile tests. Not for deciding what to test or for merge gates. |
tensor-grep: hermetic + hostile test construction
The 2026-08-08/09 wave (M1, M3, M14, #979, #984) hard-confirmed two construction laws that keep
costing rounds when skipped:
- A hostile fixture is a claim about the world, and claims get verified. A test whose
permission-denial / junction / symlink-swap setup silently no-ops does not test anything — it
reports a clean PASS on a fixture that never applied (oracle Form 6, AGENTS.md #281).
- A gated test that must pass in BOTH the dev env AND CI pytest envs must be env-independent BY
CONSTRUCTION (A85, #984): force a deterministic seam for the optional engine, never
env-detect; a test that passes locally and fails CI on a missing engine is a DEFECT in the test,
not the product.
The two laws are one discipline: a fixture (or a test environment) is evidence only once you have
independently proved it is the hostile thing you claim it to be.
Part 1 — Hermetic construction (env-independent BY CONSTRUCTION)
1.1 Force a controlled deterministic seam — never env-detect
CI's pytest env lacks SOME of the optional engines the dev desktop has — but not all of them; know
which is which before forcing a seam. A gated test that reaches a tool's success arm through a real
engine flips its verdict between the two environments:
| Optional engine | Dev desktop | CI pytest env (test-python) | Flip |
|---|
| ast-grep binary | present | absent (installed only in the agent-readiness job, not test-python) | AST-wrapper tool success arms → absent-dep raise / "unavailable" envelope |
| tree-sitter native grammars | present | PRESENT — test-python installs -e ".[dev,ast]" and the ast extra carries every grammar | none — do NOT force a seam for these |
| dense model (model2vec) | present/corrupt | absent (semantic extra not installed) | tg_find dense → BM25 fallback |
compiled rust_core Python extension | built | PRESENT — the maturin build backend compiles the PyO3 cdylib as part of the editable install | none — extension-backed probes run |
STANDALONE native tg binary (rust_core/target/release/tg) | built | may be absent — test-python never deliberately builds it (maturin builds only the extension cdylib); claim IN DISPUTE, see the non-gating Task 22 diagnostic step in ci.yml | delegation-route probes → Python route |
(Verified 2026-08-12 against ci.yml test-python install steps + pyproject.toml extras/build
backend; grep uv pip install -e ".[dev,ast]" in .github/workflows/ci.yml and ast = /
build-backend in pyproject.toml to re-derive.)
The mechanism (M14 census shape, _AST_ENGINE_SHIM_FAMILIES in mcp_server's M14 test): force
the engine's absence path through a controlled shim whose __name__ is load-bearing (the tool
refuses a backend not named AstBackend), returning deterministic empty results — so the tool's
REAL success return site (e.g. _inject_mcp_contract_fields, src/tensor_grep/cli/mcp_server.py)
is value-checked identically everywhere. For the dense leg, the ratchet does NOT raise
DenseUnavailableError: _force_dense_unavailable in
tests/unit/test_mcp_contract_stamp_ratchet.py REPLACES retrieval_dense.dense_available with a
deterministic (False, reason) tuple, so every venue exercises the same BM25-only fallback success
path (the same answer the degrade writes into rank_fallback_reason).
Checked list for a hermetic gated test:
1.2 Mutation-control: a ratchet that cannot RED is decoration
Before trusting any census/ratchet-style gated test, prove it goes RED on each mutation (Form 1
applied to guards, AGENTS.md), and re-point the ratchet until it does:
The M14 sequence was the receipt: the ratchet (not the fix) took two extra codex rounds, and each
round found a harness defect (exception-allowlist masking, env-dependence on the dense model,
partial-key parity).
1.3 Platform arms: target AND non-target (A84)
A gated test that exercises a platform-meaningful path shape (a Windows junction, a drive-absolute
strip, a /mnt/c bridge) must pin BOTH arms: the TARGET platform where the shape is meaningful AND
the NON-target platform where the transform must be inert (A84). An unconditional platform-shaped
transform re-creates the escape on the sibling OS — the drive-absolute strip receipt: stripping the
leading / from /C:/… unconditionally turns a root-anchored URI into a RELATIVE path on POSIX,
flipping a confinement check from refused→passed. Gate the transform on os.name == "nt" (or its
POSIX analogue) and pin both arms in a cross-platform test; the junction fixture of Part 2 is the
same shape (the Windows junction arm plus a sibling symlink arm for the POSIX topology, neither
covering the other — the A27/A39 twin rule). The real CI OS matrix is the only oracle that catches
a platform flip; a single-platform local green proves nothing about the sibling arm.
1.4 Patch the DEFINITION site — a namespace patch over a function-local import is INERT (2026-08-21)
monkeypatch.setattr(some_module, "name", ...) only works if the code under test READS
some_module.name. When the production path does a function-local import, the name is resolved
from the SOURCE module at call time and the attribute you set is never read. The test then runs on
whatever the machine actually has — the exact environment dependence the fixture exists to remove.
raising=False is what makes this silent. It exists to allow setting an attribute that does not
exist yet, so it suppresses the AttributeError that is otherwise your only warning that nothing
reads that name.
def _run_find(...):
from tensor_grep.core.retrieval_dense import dense_available
available, reason = dense_available()
monkeypatch.setattr(cli_main, "dense_available", lambda: (True, None), raising=False)
monkeypatch.setattr("tensor_grep.core.retrieval_dense.dense_available", lambda: (True, None))
Measured (PR #1087). With the inert form, forcing available=True and available=False
produced an identical routing_backend — the same arm twice, wearing two names, across three
tests that all passed. Positive control in the same probe: patching the definition site MOVED the
payload, proving the value could change and the fixture simply was not reaching it.
The two-part proof any forcing fixture owes. Passing tests do not demonstrate either half:
- BINDS — a sentinel value set by the fixture appears in the observable output.
(
rank_fallback_reason == "semantic ranking unavailable: forced off for this test".)
- DISCRIMINATES — that value DIFFERS from the unpatched run.
(Unpatched:
"...model2vec not installed...".)
Second-order cost: a namespace patch can re-weld a Route A target.
scripts/bare_call_ratchet.py counts calls to names the SUITE PATCHES on a module — so the
offender set is defined by the tests, not by the source. Adding setattr(cli_main, "dense_available", ...) turned three pre-existing, untouched bare dense_available() calls in cli/main.py into
UNPINNED OFFENDERs and failed CI, on a file the diff never touched. Two misdiagnoses to skip:
- "the rebase clobbered the pins file" — the pins were byte-identical on
main, and on both
branches. Diff them before theorising.
- "this is pre-existing on main" — the call sites were, at identical line numbers. The patch
was new. Compare the patch set, not the call sites.
Its message the pins file is empty but targets still have bare calls -- the gate is off reads as a
broken gate; bare_calls: {} is correct (all three Route A targets converted to 0). The gate was
on and right.
Tell: a fixture parameterised True/False whose two arms produce identical output, and a
raising=False beside a name the module does not define.
Part 2 — Hostile-fixture construction (BITE it, or it bites you)
2.1 The fixture-BITES precondition (Form 6, AGENTS.md #281)
A hostile fixture — permission denied, network partition, disk full, killed process, corrupted file,
missing binary, junction/symlink swap — is a claim about the world. Before the probe runs,
assert the fixture actually delivered the hostility, and abort loudly if it did not:
try:
os.listdir(denied_dir)
raise AssertionError("fixture is vacuous: entries listed -> the deny did not apply")
except PermissionError:
pass
The icacls-fails-twice receipt (#281) is why this is non-negotiable: a deny ACE that does not
apply leaves the directory perfectly readable, tg returns a complete result, and the honest-looking
conclusion is "no defect — the payload is intact." The bug is declared absent by a test that never
tested anything. A fixture is a claim about the world, and claims get verified.
2.2 The repo's canonical hostile-fixture: Windows junctions on the M1 checkpoint guard
tests/unit/test_checkpoint_create_ancestor_confinement.py is the in-repo pattern for BOTH halves
of this discipline (M1, #982). Windows specifics that matter — a junction is NOT a symlink: Path.is_symlink() is False on a junction. SUPERSEDED for the pinned Rust 1.96.0 toolchain: a real mklink /J junction reports is_symlink: true / is_symlink_dir: true / is_symlink_file: false (bounded probe receipt: docs/design/2026-08-13-replace-in-place-symlink-threat-model.md section 5); the CPython os.path.islink() half of the claim stays true.
- Junctions (
mklink /J on Windows, or New-Item -ItemType Junction) need NO privilege, unlike
symlinks — so Windows is the attack platform you MUST exercise, not skip.
Path.is_symlink() is False on a junction; os.walk/os.scandir descend junctions as plain
directories. SUPERSEDED for the pinned Rust 1.96.0 toolchain: a real mklink /J junction reports is_symlink: true / is_symlink_dir: true / is_symlink_file: false (bounded probe receipt: docs/design/2026-08-13-replace-in-place-symlink-threat-model.md section 5); the CPython os.path.islink() half of the claim stays true. So the enemy property is "an ANCESTOR directory that resolves out-of-root", not
"a leaf link".mklink /J requires the LINK path to NOT pre-exist; the TARGET may be fully populated.
SUPERSEDED (2026-08-12 — was the A88 dogfood wording): this bullet previously said mklink /J
"silently fails to create a junction when the target directory is NON-EMPTY". That claim is
WRONG, verified 2026-08-12 two ways: (1) an empirical probe on this host — mklink /J against a
target CONTAINING a file succeeds (exit 0, the junction resolves through to the file), while
mklink /J onto a PRE-EXISTING link path fails (exit 1, "Cannot create a file when that file
already exists", which reads as "silent" only when the checker captures output and looks at the
exit code alone); (2) the repo's own fixture helper _plant_ancestor_link_or_skip
(tests/unit/test_checkpoint_create_ancestor_confinement.py) removes the LINK path first
(shutil.rmtree(link)) and then junctions to a target that CONTAINS b.txt — the fixture works
precisely because a populated target is fine. The A88 dogfood fixture that "never applied" is
therefore best explained by the link path still existing at creation time, not by a populated
target (AGENTS.md's A88 entry still carries the old wording — out of scope for this skill, but
do not re-cite it as authority for the non-empty-target claim). The BITE precheck stays
mandatory regardless: assert the redirect actually resolves, AND assert the negative shape
(assert not link.is_symlink() (for CPython pathlib) so a real symlink is not mistaken for a junction). SUPERSEDED for the pinned Rust 1.96.0 toolchain: a real mklink /J junction reports is_symlink: true / / (bounded probe receipt: docs/design/2026-08-13-replace-in-place-symlink-threat-model.md section 5); the CPython half of the claim stays true. See
plus the inline BITE precheck in each test body (grep
) in that test module — there is no /
helper (was: cited here as / ; now: the real symbols).
Checked list for a junction/symlink hostile test:
2.3 Symlink-ancestor + junction-ancestor are DIFFERENT fixtures
M1's plan (docs/plans/2026-08-08-backlog-completion-plan.md, M1 section — grep ### M1) and the
test module both treat the two separately: a leaf symlink is legitimately tracked and stored AS a
link, while a symlinked OR junctioned ancestor under root is refused because the OS traverses it
transparently and copies out-of-root content. Write a distinct fixture per topology; do not let one
passing arm "cover" the sibling (the A27/A39/twin rule) — and note the create-side guard
(checkpoint_store.py _resolve_parent_within_root, parent-chain-only resolve) and the undo-side
full-leaf resolve (_resolve_within_root) are separate contracts; see 2.4 for the seam receipt
(was: this paragraph cited the create-side guard as _resolve_within_root at :149 — that is the
UNDO-side symbol with a bare line number; now: both symbols cited by name).
2.4 Event-gated parent-swap fixtures (A38/A48) — the TOCTOU a plant-once fixture cannot reach
Planting a link ONCE (2.2/2.3) proves a STATIC out-of-root resolve. It cannot prove the swap race:
an attacker swapping a PARENT directory or junction BETWEEN the containment check and the
create/lock/publication step (A38). The construction law for that stronger fixture:
- Anchor to OPENED, identity-verified parent handles. Resolve-then-act over path strings is
the SHAPE under test, never the oracle — see
tensor-grep-cross-platform-path-confinement
Part 3 (Canonicalize-or-fail-closed vs opened-identity anchoring, A38/A48/A53).
- Event-gate the swap (never wall-clock overlap — a starved runner serializes legitimately and
false-fails; assert the blocking CONTRACT with
threading.Event handshakes, AGENTS.md A17/A27):
swap the parent deterministically AFTER the guard's check/lock and BEFORE the step under test, in
each of the three windows A48 names — before create, after lock, before publication/read.
- Repo seam (verified 2026-08-12, cite by symbol): the CREATE-side guard is
checkpoint_store.py::_resolve_parent_within_root (parent-chain-only resolve; the leaf's raw
identity survives — called from create_checkpoint's copy loop), and the UNDO-side twin is
_resolve_within_root (full-leaf resolve over the snapshot dir). Grep
def _resolve_parent_within_root / def _resolve_within_root in
src/tensor_grep/cli/checkpoint_store.py to re-derive.
- Honest current state (A48 DEFERRED): the atomic-publish helper
atomic_write_bytes_anchored
(src/tensor_grep/cli/_index_lock.py) is FSYNC-anchored (data fsync + directory fsync,
O_NOFOLLOW temp creation, leaf-symlink refusal) but NOT identity-anchored — it does not yet
open/verify parent handles. Until A48 lands, a full Event-gated parent-swap RED against the
publish path stays RED-by-design: the TOCTOU is real and unclosed, so the RED is the correct
expected state, not a fixture bug. Write the fixture, mark it RED-by-design with the A48 owner
and reopen trigger, and do NOT relax the assertion to green.
Part 3 — Mutation asserted-applied (the red-arm must actually RED)
Behavioral-RED law (A61): a RED arm must fail for the EXACT expected refusal/reason class. A
crash, import failure, hang, panic, or setup error is NOT a behavioral RED — it proves the arm
died BEFORE exercising the contract, and any arm that dies early invalidates the red phase
(AGENTS.md A61). Pin the expected exception type / message / exit code in the assertion
(pytest.raises(ExpectedError, match=...), an exact exit code, an exact refusal marker), and
reject any arm whose failure output does not contain the pinned reason. "Any error is fine"
allowlists are the same mask here as in Part 1.1's typed-allowlist rule — there on the green arm,
here on the red one.
A red-arm attempt that silently no-ops is a control arm that never ran (validation-and-qa, "A
mutation that does not apply is a control arm that never ran"):
Part 4 — Positive controls (a zero must be able to be non-zero)
Every probe carries a positive control — a zero means "measured nothing" or "never actually
checked", and the two are indistinguishable in the number (verification-oracle family, AGENTS.md):
External anchors (Exa research, 2026-08-09)
The hermetic/hostile construction discipline is this project's instance of a well-known family of
"validating the validator" practices; the family validates the shape, ours is tuned to this repo's
evidence bar (real file:line-cited receipts, red-green controls, CI-parity gating).
| Anchor | Their point | This skill's mapping |
|---|
| AIarch "Validating the Validator" (aiarch.github.io) | The check that validates a check must itself be tested; a validator is only as good as the adversarial cases it demonstrably prevents — evaluate the gate, not just the feature. | Part 1.2's mutation-control (each mutation must RED) and Part 4's positive controls: proof that the gate can fail is the only proof it works. |
| Atakua "Adaptive Deep Learning: Why We Should Care About 'How to Test the Tests'" (atakua.com.br, 2018) | Test the tests: even an obviously-useful technique must prove the tests actually catch what they claim, because an un-failing test is indistinguishable from a test that tests nothing. | Part 1.2 (mutation REDs) and Part 2 (fixture-BITES): the test's own hostile setup is asserted to be hostile before it is trusted. |
| Stack Overflow "How to test the test?" / testing-test-conf threads* | Peer review of tests, coverage of the test's own claims, and proving the red-arm fires before green. | Part 3 (mutation asserted-applied): the red phase must be observed, not assumed. |
Repo receipts to cite by symbol, not line (the never-re-stamp law): AGENTS.md oracle Form 6
(grep Form 6), A85/A88 in AGENTS.md's A-law list; tests/unit/test_checkpoint_create_ancestor_confinement.py
(the junction BITE pattern); tensor-grep-validation-and-qa SKILL.md's "Writing a hostile fixture
(Form 6 defence)" and "A mutation that does not apply is a control arm that never ran" sections;
docs/plans/2026-08-08-backlog-completion-plan.md M1/M17 sections (the fixture-BITES precheck
requirements — grep ### M1 / M17).
Quick reference
[1] env-independence force the optional-engine seam (never env-detect) -> prove the force
[2] fixture-BITES assert the hostile setup actually applied before the probe; abort if not
[3] junction != symlink is_symlink()==False on a junction; mklink /J needs the LINK path absent (target may be populated) **SUPERSEDED for the pinned Rust 1.96.0 toolchain: a real `mklink /J` junction reports `is_symlink: true` / `is_symlink_dir: true` / `is_symlink_file: false` (bounded probe receipt: docs/design/2026-08-13-replace-in-place-symlink-threat-model.md section 5); the CPython `os.path.islink()` half of the claim stays true.**
[4] mutation REDs delete a member / stamp / typed allowlist entry -> each must RED
[5] mutation applied assert the red-arm mutation actually took (diff / grep) before trusting red
[6] positive control the same probe must return non-zero somewhere it should
[7] RED reason class a crash/import/hang/panic/setup error is NOT a RED; pin the exact refusal (A61)
[8] platform arms pin the target AND the non-target platform arm; the CI OS matrix is the oracle (A84)
[9] parent-swap TOCTOU Event-gate swaps against opened identity-verified parents; the fsync-anchored publish stays RED-by-design until A48
The endpoint: a gated test that passes identically on the desktop and CI, whose hostile setup is
proven hostile, whose red arms are proven red — a test that cannot certify a lie.
