probe-input-integrity

# Probe Input Integrity — Never Fabricate an Identifier You're About to Test Against Promoted from the always-loaded `~/.claude/rules/never-fabricate-test-identifiers.md` rule, whose stub keeps the gate lines. The body below is that rule's text. When probing a system — an API endpoint, a URL, a record id, a file path, a ticket number — **obtain the identifier from the system itself. Never invent a plausible-looking one.** A fabricated identifier turns every subsequent observation into a two-variable experiment ("is the endpoint blocking me, or does this thing simply not exist?"), and the two failure modes are routinely indistinguishable in the response. ## When to Use This Skill | Use this skill when... | Use something else when... | |---|---| | About to probe an endpoint, URL, record id, path or ticket number | An empty *search* result is about to gate an action → `tool-result-traps` | | Building a harness around code that is awkward to run directly | A named entity in an error may be a zero/default sentinel → `code-quality-plugin:debugging-methodology` | | Stubbing a CLI on `PATH` for a local test | Verifying premises before a wave dispatch → `verify-before-plan` | | Your own target set is empty, so a sweep would be green by construction | | ## The trap The fabricated value *looks* real, so it never gets questioned. Its failure then gets attributed to the hypothesis actually under test. > Canonical break (2026-07, a research repo): investigating whether Reddit's > `.json` endpoint still worked, I made up a thread URL — > `/r/LocalLLaMA/comments/1i6r9rp/deepseek_r1_is_now_available_on_azure` — a > plausible id and slug, entirely invented. It returned `403 Blocked`, as did two > further probes against the same fake thread. I reported the endpoint as > IP-blocked. The user said "it works in my browser," and the *first* successful > request revealed the truth: the endpoint returned a JSON **404** — the thread had > never existed. The real discriminator was the client fingerprint, not the IP, and > three "conclusive" probes had been testing a nonexistent resource. The tell that should have fired: `403` and `404` mean very different things, and I never established which I'd get for a *known-good* id. ## The rule **Get a real identifier from the source before probing. List before you get.** ``` # Wrong — invent a plausible thread, then test against it curl https://www.reddit.com/r/X/comments/<made-up-id>/<made-up-slug>.json # Right — pull a real one out of the system first, then probe that curl -s https://www.reddit.com/r/X/ | grep -o '/r/X/comments/[a-z0-9]*/[a-z0-9_]*/' ``` Generally: a listing/search call before an item call. `gh issue list` before `gh issue view <n>`; `Glob`/`ls` before `Read`; a collection endpoint before an item endpoint. **Always run a known-good control.** When a probe fails, repeat it against an identifier you *know* exists. Without a control you cannot separate "access denied" from "not found" from "wrong shape" — and each implies a completely different fix. ## Don't fabricate the *subject* either — extract the code, don't retype it The same trap one level up. When you build a harness to exercise code that is awkward to run directly, **a retyped copy of that code is not that code**. It reads as identical in the transcript, and when it misbehaves the failure is attributed to the code under test rather than to your transcription. > Observed 2026-08 (a ComfyUI node-pack fleet): verifying new `just prs` / `just merge` > recipes over a 14-repo fleet that happened to have **zero** open PRs. The > harness retyped the kind comparison as `[ "$k" = "'release'" ]` — literal > quotes, matching nothing — and reported `PLAN=0` for every kind. That is > precisely what "no PR qualifies" looks like. Extracting the shipped loop > straight out of the justfile with `sed` and re-running produced the correct > per-kind plans and skip reasons. - **Extract the shipped text**, e.g. `sed -n '/start/,/end/p' <file>`, and feed it the inputs. Never hand-paste the logic under test into the harness. - **When your own target set is empty, borrow a control set.** A sweep over zero items is green by construction and asserts nothing. Point the extracted code at real objects from elsewhere that cover every branch — for the recipes above, live PRs from other repos spanning release (bot- *and* human-authored) / deps / mine / external, plus draft, `BLOCKED`, `UNSTABLE`, and failing-checks. An empty run and a broken harness produce the same output. Only the control tells them apart. ## Don't fabricate the *environment* either — an inert stub means the real tool ran The third form. The subject is real and the inputs are real, but the harness never took effect: a stubbed CLI placed on `PATH` that is **not executable** is skipped by PATH lookup in silence, and the real binary runs instead. Nothing warns. The run completes, prints plausible output, and — if the real tool has side effects — performs them. > Observed 2026-08 (an infrastructure repo): testing a workflow step that posts > a PR comment, I wrote a fake `gh` onto `PATH` with the Write tool, which does > not set the executable bit. The real `gh` ran, and what was framed as a local > test **POSTed and then PATCHed a comment onto a live PR**. It surfaced only > because the output carried a real comment ID where the stub's canned `12345` > was expected. Re-running with `chmod +x` behaved correctly. - **Set the bit and check it.** `chmod +x <stub>` via Bash after any Write, then `ls -l`. File-creation tools generally do not set the executable bit. - **Assert the stub is the one in effect before trusting any case.** `command -v <tool>` at the top of the harness, or better, have the stub print a sentinel the real tool never would and fail the run if it is absent. - **Make the stub's output impossible to mistake for the real thing** — a canned `12345` beats a realistic-looking id, so a discrepancy is visible at a glance rather than plausible. - **Treat "it's only a local test" as a reason for more isolation, not less.** That framing is what suppresses the caution normally applied to a command that writes to shared state, and whether the writing tool runs is precisely what is in question. Same law as the two forms above, one layer out: the harness was green while measuring production. ## When it bites - Diagnosing whether an endpoint, credential, or permission works — exactly where a wrong conclusion triggers a redesign that was never needed. - Writing tests or repro cases against "example" ids nobody verified exist. - Citing an issue/PR/commit number from memory instead of looking it up. A plausible-but-wrong `#162` is worse than no number: it reads as researched. ## Relationship to sibling rules - `verify-upstream-before-patching.md`, `verify-machine-facts-before-publishing.md` — same instinct (consult the authoritative source before acting). This skill covers the *inputs to your own diagnostics*, not the code you patch or the facts you publish. - `tool-result-traps` and `harness-tool-errors` (Read on a missing path; the `rg -r` fabrication trap) — siblings in the same family: a confidently-wrong output that looks exactly like a real one. ## Rationale An invented identifier is a silent confound. It costs one extra lookup to obtain a real one, and one extra probe to run a control; skipping both bought a wrong root cause, a confidently-reported non-finding, and a diagnosis that unravelled only because a human said "but it works for me." The model will not catch this on its own — in the transcript, a fabricated value looks exactly as legitimate as a real one.