verify-go

name

verify-go

description

Before COMMITTING or REPORTING a positive result — a GO, a "surpass", a milestone, an "it works / it's inert / byte-identical / tests pass" — run independent ADVERSARIAL skeptics that each try to REFUTE it from a distinct angle, then commit only what survives (commit-with-caveat, never silently drop, if a mischaracterization is found). Use whenever a result would change a wall/gap status, flip a default, or land in a findings doc / the board. This operationalizes drift-mode

Verify-GO — adversarially verify a positive result before it lands

A GO that looks clean is the most dangerous thing in this project: it gets committed, flips a default, and mis-aims the next session. The 2026-07-24 session produced four over-claims caught only by adversarial verification (W3 "structural immunity", P1.2 affect-scalar, 2 gap#5 lucky-draws) and six silent-failure retractions — three self-authored. Vigilance doesn't catch these (I authored a bare-except swallowing my own warning the day I documented that pattern five times). A procedure does.

Announce at start: "Running verify-go: adversarially probing this result before it lands."

When to run

• Before COMMITTING a finding that reports a GO / surpass / milestone / capability close.
• Before FLIPPING a default or claiming a cheat/shortcut is closed.
• Before writing "inert / byte-identical / no-op / tests pass / pushed / on GPU" as fact.
• Before recording even an INTERIM "lead" / "partial positive" / "realizable" / "first non-flat result" into a finding or the board (2026-07-25: a "lead" I wrote into TWO docs — "the two-sided read is REALIZABLE, fact-1 own/other 3.67" — was a winner-slot artifact; the disambiguating control, run only later by an adversarial subagent, refuted it. A non-flat number is not a lead until its control collapses; an interim positive mis-aims the next step exactly like a GO does).
• NOT for routine mechanical edits (a typo, a refactor with a passing test you watched pass).

The procedure — dispatch independent skeptics, one per lens

Spawn N skeptics (a Workflow parallel stage, or concurrent Agents) — each gets the result + the runner/finding and is told: "Try to REFUTE this from the angle. Default to REFUTED if uncertain." Independence matters: redundant identical skeptics miss failure modes diverse lenses catch. Assign distinct lenses:

1. Reproducibility / power — does it hold at 6 seeds (42/43/44/100/101/102)? Is the effect bigger than seed-to-seed noise? A 3-seed indicator is not a GO. Is one lucky seed carrying it?
2. Gate-cheat — can the gate PASS without its key control? Is the anti-cheat control DEFAULT-ON and actually INVOKED (grep the call site), not just defined? A gate that passes without its control IS the bug.
3. Control-integrity — is the A/B lever ONE variable? (One flag ≠ one variable — a global clamp touched both a spiking synapse and a host readout.) Does the DEFAULT arm genuinely differ from the treatment (print the lever's effect; if both arms froze, the verdict is void while looking plausible)? AND: are BOTH arms pinned at the FLOOR or the CEILING? A comparison between two saturated arms has ZERO discriminating power and cannot detect a difference that exists — the resulting "no difference" is VOID, not evidence. Always print the raw magnitudes of both arms and confirm they are in a responsive range before believing any null. (2026-07-25, TWICE in one session, opposite ends: a learning rate above the saturation knee drove every synapse into the soft bound and pinned a real selectivity signal at a flat ~1.0 — the signal appeared as soon as the rate came down; and a "did this regress?" A/B was run at a training budget where BOTH old and new code sat at 0/16, producing a confident "nothing regressed" that did not follow from the test at all, and had to be retracted once the un-floored budget showed 12.5% vs a recorded 87.5%.)
4. Instrument-trust — read the runner's OWN verdict line; NEVER lift a metric from a run that printed SIGNAL=False / HONEST NEGATIVE. Is the metric quantized/rounded so the effect is unfalsifiable by construction (differencing round(x,4) values)? A refutation needs the instrument verified exactly as much as a confirmation.
5. Seeding — is cfg.seed set (NOT actual_seed_used, which seeds nothing)? Build twice at one seed and hash cp_neuron_firing_thresholds — identical ⇒ actually seeded.
6. Infra — pushed? verify with tools/push_both.sh (ls-remote, not echo pushed). On GPU? tools/monitor_runs.py reports [GPU]/ON CPU. "byte-identical when off"? that's an ASSERTION (a test), not a comment.
7. Selectivity-metric bias — the MASS artifact (three instances in one session, 2026-07-25; treat as the default suspect for ANY ratio). For ANY "own-vs-other" / ratio / argmax-selectivity metric (own/other, recall picks the right slot, this cue selects that assembly, this window is more specific), the null hypothesis is — more weight, more spikes, more cells. Three checks, all mandatory, all cheap:

Not every lens applies to every result — pick the ones with teeth for this claim, but a GO that flips a default earns at least reproducibility + gate-cheat + control-integrity.

Synthesize + act

• Survives all lenses → commit as a GO. Say which lenses were run (the verification is part of the deliverable).
• A lens finds a mischaracterization → do NOT silently drop or quietly downgrade. Commit-WITH-CAVEAT: state precisely what the result IS vs what it was claimed to be (W3 was a real GO but not "structurally immune"; the affect wiring was real but not the scalar claimed). An honest narrowed result is a first-class deliverable.
• A lens refutes it → retract the GO, bank the method as a NO-GO with the root cause (per THE LAW the capability stays open), take the next mechanism.

Verifying a NEGATIVE — before you accept a BOUNDARY (2026-07-25: the costliest error of the session)

A wrong GO wastes a build. A wrong NO-GO closes a capability that was never actually blocked — and it is far harder to notice, because a negative feels like rigour. Before recording any "wall / boundary / characterized limit / the substrate can't", run these:

1. IS THE SUBSTRATE PHYSICALLY VALID? Read its state variables in their PHYSICAL UNITS and check them against physiological range. Membrane potentials belong in ≈ −90…+50 mV; conductances, currents and rates have known scales. No metric can catch this — only looking at the numbers in units can. (2026-07-25: an entire multi-hour "dense CA1 code ⇒ no write can localize" boundary — ~15 write variants, a sparsification battery, a two-sided read, two subagent builds, several confident findings — was a 333× miscalibration of a pA→mV constant in ONE config line. v_apical was parked at ~2×10⁵ mV, driving every soma; the "dense code" was runaway current, not a hippocampal code. At a valid operating point the real code is sparse and near-disjoint and the write localizes, 6-seed.)
2. Check every constant that was TUNED while the artifact was present. They were fitted to broken dynamics and will silently re-break the fixed system. (Same session: a cell phenotype adopted to fight the artifact was undrivable once voltages were physiological; a learning rate fitted to ~100× inflated activity saturated the write and pinned the metric at exactly the null being "confirmed".)
3. Is the measurement itself inert? A measurement must never be plastic. If learning is enabled while you read, you are measuring the reader. (Same session: core sizes varied with the write learning rate although cores are defined pre-write — because the rule was still learning during the read.)
4. Does the null survive at more than one operating point? A null measured at a single point is a property of that point. Sweep the parameter the mechanism is most sensitive to (here: the learning rate through its saturation knee) and confirm the null is flat across it — a monotonic trend toward signal means you measured a bad operating point, not a bound.
5. Do the positive controls fire? If your harness cannot demonstrate the effect where it MUST exist, the harness is what you have measured.
6. CITING A LINE IS NOT VERIFYING IT EXECUTES. Find the guarding flag and check its DEFAULT against the config in use. A grep hit proves code EXISTS; it proves nothing about whether it RUNS — and a file:line citation makes an unverified claim look verified, which is worse than no citation. (2026-07-25: I reported a mechanism as "the whole story", cited to sim/bridge.py:838. That line sits inside _apply_branchless_hebbian, "opt-in via cfg.enable_branchless_plasticity", default False — it never ran. The rule actually executing was a different one with different dynamics, and an entire "characterized ceiling ⇒ the next step is structural" verdict was built on the dead branch.) in the same arc defaults to and the pathway was , so STDP was writing it throughout and never entered a 7-hypothesis ledger. produced a "hard 5.0 ceiling" that was read as a result, when it was a literal reached because that arm disabled Hebbian — so the mechanism under test was never exercised at all.

Trigger: the moment you are about to write "boundary / wall / can't / characterized limit / honest negative", or a mechanism has failed across many well-controlled variants. Many variants failing identically is itself evidence of a COMMON upstream cause — the shared substrate or a shared constant — not of independent confirmations.

Verifying a DIAGNOSIS — the prescribed fix must be able to REFUTE it (2026-07-26, earned twice in one session)

A "diagnosis" (this mechanism causes that defect) is a CLAIM and gets the same adversarial treatment as a GO. Twice in one session I wrote "DIAGNOSIS COMPLETE", prescribed the fix it implied, and the fix refuted the diagnosis: (1) "a single SHARED inhibitory pool causes the winner-take-all" — built per-slot FS cross-inhibition; identical to the shipped global pool in all three conditions. (2) "the hebbian_max_weight inversion causes it" — re-ran with the bound above the init; the winner persisted. One of these was independently corroborated by a read-only research gate — which was ALSO wrong, because it reasoned from the CODE, not the DATA.

The rules:

• Always build the fix WITH its lesion arm (the unfixed topology/config), in the SAME run. If fix ≈ lesion, the DIAGNOSIS is refuted — not the implementation. Without the lesion you will read "still broken" as "fix was too weak" and tune forever.
• Test the fix in the condition where the defect APPEARS. I first compared topologies with the gaps unfrozen, where uniform potentiation masks any topology effect — a null that meant nothing.
• Corroboration from code-reading is not evidence about behavior. A second reader agreeing with your mechanism story raises confidence in the READING, not in the CAUSE. Only a manipulation does that.
• Prefer a MEASUREMENT that separates the candidate causes over another prescribed mechanism. After 11 refuted hypotheses, what finally settled it was instrumenting the quantity itself (per-window dw), which showed the write was near-symmetric and the outcome a ~3% residual — a shape NONE of the 11 hypotheses predicted. When two successive mechanisms are refuted, stop prescribing and go measure.
• A rate lever cannot move a FIXED POINT. If the state settles at a soft bound (dw ∝ (w_max − w)), every learning-rate sweep is inert by construction — it changes how fast you reach the bound, not where it is. Before sweeping a rate across orders of magnitude, ask whether the observable is a fixed point; if it is, sweep the STRUCTURE instead. (This retroactively explains a long history of "invariant across every lever" on one pathway.)

Parallel arms that vanish are a SILENT FAILURE, not a result (2026-07-26)

A for … & done; wait fan-out of GPU runs silently dropped 3-of-4 and then 3-of-3 arms (VRAM contention), exiting 0 with empty output. The surviving arm looked like a clean single data point. Rules: echo each arm's ${PIPESTATUS[0]}; treat "an arm produced no output" as a FAILED RUN to reproduce, never as a null result; and when arms contend for one GPU, run them SERIALLY — a slower correct sweep beats a fast one whose failures are invisible.

⛔ THE RESEARCH GATE HAS A LOOPHOLE: a SEQUENCE of cheap tests is a BUILD EFFORT (2026-07-26)

CLAUDE.md fires the research gate before "committing ANY build / GPU / sim/-edit effort to overcome a difficulty". I evaded it for hours without ever deciding to — because each individual step was a cheap config flag, never "a build". I tested weighted-vs-count coincidence, the self-regen latch, the Hebbian bound, inhibition topology, an inter-window washout, and cue magnitude — six levers, ~4 GPU-hours, all against ONE defect — and the gate never subjectively "fired" because no single test felt like a commitment. When I finally dispatched the research round, it resolved in one pass what the sequential guessing had not.

MECHANICAL RULE (no judgment call, matching the gate's design intent): if you have tested ≥2 distinct levers against the same defect without resolving it, the gate FIRES. Cheapness of each individual test is not an exemption — the relevant quantity is cumulative effort against one difficulty, not the cost of the next step. Write the lever count in the findings doc so the counter is visible.

Self-check: "am I about to try a third thing against the same failure?" → that IS the trigger. Dispatch the read-only research round; your candidate becomes one ranked option, never the default.

⛔ AND THE ≥2-LEVER TRIGGER IS NOT ENOUGH — THE CHEAPEST GUARD RUNS BEFORE THE FIRST LEVER (2026-07-26). When the research round finally ran, its first finding was that a 497-line research gate for the identical defect, on the identical substrate, with a ranked 6-mechanism ladder, was 2 DAYS OLD — and its predecessor had already written the failure down verbatim. A day of lever-chasing re-derived it. It also showed my "5th instance" framing was wrong (8th in the project, a documented FAMILY ⇒ gate condition (b) fires on FIRST occurrence) and that 3 of my "new" measurements were 4th independent confirmations of results already in the corpus.

MANDATORY, ~30 SECONDS, BEFORE THE FIRST LEVER AGAINST ANY DEFECT:

.venv-rag/bin/python tools/rag/rag_search.py "<the defect in one line>" 5 --corpus finding

Ask literally: has this already been scoped, tried, or refuted? Then READ any research-gate / scope doc it surfaces before touching a flag. Cost: one query. Benefit here: a day. This is drift #12 (the stale/skipped corpus) in its most expensive form — not acting on a stale summary, but never asking whether the answer already existed.

Measure the thing, at the time it happens (2026-07-26 — retraction #9)

Three separate ways a measurement can be structurally incapable of answering its own question, all seen in one sub-arc:

1. Placed upstream of the effect. A per-fact weight table ran BEFORE the coactivation_replay it claimed to characterise; a "BOUNDARY LOCATED" finding was committed on numbers that could not have shown a replay effect. Print/measure the quantity BEFORE and AFTER the manipulation, and report the delta — then a misplacement is visible as a zero delta instead of a confident wrong table.
2. A lesion that does not persist. Zeroing cp_connections.data survived one step and regrew (plasticity was live). Re-read the manipulated quantity at the moment of measurement, not when you issued it.
3. A metric too coarse to resolve a real lever. An argmax decided by 400-vs-0 cannot move on a 1.7% change. Always carry one CONTINUOUS quantity alongside any count/argmax metric.

⇒ A NULL A/B HAS THREE EXPLANATIONS — inert lever · misplaced measurement · coarse metric — and they are indistinguishable from the summary alone. I misread "byte-identical arms" as "inert lever" TWICE in one session, wrong for a different reason each time. Distinguish them by measuring the lever's effect on a continuous quantity before interpreting the outcome.

"CORRECTLY DID NOTHING" MUST NOT PAD THE HEADLINE (2026-08-19, from Shiu & Sterne et al. 2024)

When a lesion / screen / anti-cheat sweep is DOMINATED by cases where the model CORRECTLY predicts NO effect, those easy true-negatives inflate the headline accuracy. Report the score BOTH with AND without the correct-silence cases, and LEAD with the harder number. Precedent: Shiu & Sterne et al. 2024 (Nature 634:210) reported 91% of predictions matched, and SEPARATELY 84% excluding the split-GAL4 screen where most cell types correctly did nothing — the 84% is the honest headline. Here: whenever a sweep's "collapse"/lesion arms are mostly expected-zero, state how many cases are correct-silence and give the score net of them, so a wall of trivial true-negatives cannot read as a strong result. Pairs with the single-seed / headline gates.

What this skill MUST NOT do

• Rubber-stamp — a skeptic that "confirms" without trying to break it did nothing. The prompt must push to REFUTE.
• Verify only the happy path — test the claim against the case you'd EXPECT to break it (a run you KNOW is broken, the seed you fear).
• Treat a refutation as needing less scrutiny than a confirmation (rule 3 of the SILENT-FAILURE CLASS).
• Touch the science verdict itself dishonestly — this skill SHARPENS a claim to the truth, it never launders a weak result into a strong one.

Why this skill exists

2026-07-24 (evolve-skills): adversarial verification was run ad-hoc (via one-off Workflow panels) and reliably caught real over-claims — but it lived only in-session, not in a skill, so it depended on remembering to do it. This encodes it as a reflex triggered by the commit itself. Pairs with the neural-simulator skill's SILENT-FAILURE CLASS (the specific checks) — this is the procedure that runs them before a GO lands.

ASSERT THE GATE EXISTS BEFORE YOU FREEZE OR LESION IT (2026-07-29)

_try_pgate swallows the KeyError and returns False for a gate that does not exist. _mean_gate_weight returns 0.0 for one. Nothing checks either return value. So freezing a NONEXISTENT gate is a silent no-op that presents as a perfect freeze: drift exactly +0.000000.

That is not hypothetical — a gate's existence depends on config. comp_no_pool_slot=True (the value in BASE) drops the pool→slot pathway entirely, so concept_to_comp_attr is simply absent, and every read of it returns nan/0.0 rather than raising.

The rule: before any freeze / lesion / weight-read on a NAMED gate, assert it exists:

assert gate in bridge._plasticity_gate_indices_gpu, (
    "gate %r absent under this config -- freezing it is a SILENT NO-OP that reads as a perfect freeze" % gate)

Why it matters beyond the bug: an exact +0.000000 freeze-drift is the signature of a real freeze AND of a missing gate. They are indistinguishable in the log. A suspiciously perfect number is a prompt to check the instrument, not evidence that the manipulation worked. (Found while chasing an unrelated nan; the affected headline result was checked and STANDS, because its own probe sets comp_no_pool_slot=False — but only checking revealed that.)

THE ENGAGEMENT COUNTER IS THE CHEAPEST GUARD YOU HAVE — and tools/lab.py already implements it (2026-07-29)

Every void arm this project has produced shares one shape: the manipulation never engaged, and the resulting null looked like a scientific result. Three in a single day:

• a metaplasticity toy where theta0 was unreachable, so every trial fell to a fallback branch that did all the work — caught only because all six betas printed IDENTICAL maps and identical block counts;
• a substrate probe reading a gate that did not exist under its config — caught because it printed nan;
• a saturation test where the soft bound never bound (sat_frac = 0.000 in every arm), so identical numbers across w_max proved nothing — caught by the engagement fraction it happened to print.

The rule: every arm must report a number that goes to ZERO when the mechanism is inert, and you must look at it before reading any score. blocks, sat_frac, n_overrides, dw, gate n_syn — whatever the mechanism does, count it.

And USE THE HELPER — this is the actual lapse. tools/lab.py exists precisely for this and was written after the last round of void arms:

from tools.lab import lever, void_if, before_after, undefined_if_empty
lever("w_max", before=unbounded_score, after=bounded_score, continuous=sat_frac)
void_if(sat_frac == 0.0, "the soft bound never engaged; w_max arms are identical by construction")

The saturation arm above was written WITHOUT importing it, by the same person who wrote it. A helper you do not import is exactly as useful as a rule you do not remember — which is the whole reason lab.py exists. Import it at the top of every probe, not when you suspect trouble.

PARALLELIZATION IS A DISPATCHER, NOT A DECISION (2026-07-29 — the owner flagged this TWICE in one day)

The failure was not missing information. The heartbeat printed UNDER-FILLED-GPU every 15 minutes and the 36-core pool sat at load 0.00 for hours, while substrate conclusions were being drawn from a SINGLE seed against this project's documented 6-seed standard. The warning fired correctly and was acted on minimally, twice, because responding to it required inventing a job on the spot — so the cheap response was always "launch one more thing and move on".

The mechanism: tools/lane_dispatch.sh <gpu|pool> <slots> keeps N slots busy from a persistent queue file (research/queue/gpu.queue), moving each line queue → .running → .done so state survives a restart and nothing is double-run.

bash tools/lane_dispatch.sh gpu 7 &          # keeps 7 GPU jobs alive from the queue
cat >> research/queue/gpu.queue <<< "<one shell command per line>"

THE ALARM MOVED, and that is the actual fix. The heartbeat no longer warns on an idle lane — it warns on GPU-QUEUE-LOW and DISPATCHER-DEAD. An idle lane with a stocked queue self-heals in seconds; an empty queue guarantees future idleness. So the only standing obligation is keep the queue stocked, which is the existing "build de-risks ahead of time so idle compute always has a ready job" directive made executable instead of remembered.

Stock it with the work that is already OWED, not with invented work. On the day this was built the queue filled instantly from two real debts: seeds 43/44/100/101/102 for a headline contrast that had only seed 42, and the clean scale series whose confound had already been diagnosed and written down. If the queue looks empty, the likelier truth is that owed replications are being skipped.

POOL ADDENDUM (same day): the pool's under-use was NOT a dispatch problem, and a dispatcher there was the wrong fix. A pool_dispatch.sh was written, debugged through three failures (a pgrep -fc that prints 0 AND exits nonzero so || echo 0 emitted two lines and broke the arithmetic; an exec -a marker the wrapper shells also carried, over-counting 3x; a regex that silently missed a one-line function), and then deleted — because the plain xargs -P 12 per node that already worked ran a 72-cell × 6-seed sweep (432 runs) across 36 cores in under 60 seconds.

The real constraint on that lane is JOB SIZE, not dispatch. Numpy sweeps drain the pool faster than any queue can be stocked, so "keep the pool full" is the wrong goal — the right one is "send the pool work worth 36 cores", i.e. batch the whole grid at 6 seeds instead of trickling 2-seed probes. The GPU dispatcher stands because GPU arms run ~90 minutes; the same mechanism on a seconds-per-job lane is ceremony. Match the mechanism to the lane's job duration.

AN AD-HOC CHECK THAT DISAGREES WITH A VERIFIED TOOL IS WRONG UNTIL PROVEN OTHERWISE (2026-07-29)

tools/check_docs.py reported W2 clean. A one-line awk 'length>800' appeared to find five violations, and the instinct was "the checker has a gap". It did not. The awk was wrong three separate ways: it used NR (cumulative across files) instead of FNR, so its line numbers pointed into a different file; it did not exempt table rows; and it did not track code fences. Every "violation" was an exempt line.

The rule: when a throwaway check contradicts a tool that has tests and a stated specification, debug the throwaway check FIRST. The tool encodes exemptions and edge cases that a one-liner cannot. Reversing that order costs time and — worse — can produce a "the checker is broken" finding that is itself the bug.

Same session, same shape, three more times: a pgrep -fc that prints 0 AND exits nonzero (so || echo 0 emitted two lines and broke arithmetic); a margin metric read as improved when the weighting had rescaled its units; and a ; instead of && that let a commit through while the checker printed FAIL. The gate existed in all three cases and was not honoured. Prefer && over ; whenever a check gates an action, and never let a hand-rolled measurement overrule an instrumented one without debugging it.

LANE MONOCULTURE — a full lane is not a prioritized one (2026-07-29, owner-flagged)

After parallelization was fixed with a dispatcher, the GPU ran at 100% with a stocked queue for hours — and every job in it served ONE roadmap lane (H · Memory) while lane F · gap#4, which the master roadmap calls "the single load-bearing dependency (the crux the whole roadmap pivots on)" and "the must-solve core", had zero allocation. Not deprioritized after consideration — never considered. Simultaneously lane E · Language, tagged [CPU] and "disjoint from A/B/C", sat unqueued while the 36-core pool idled; the first lane-E runner dispatched returned a GO in 40 seconds.

The dispatcher made the lane efficient without ever checking it was the RIGHT lane. That is the failure this rule exists for: a full queue and a busy GPU look exactly like good prioritization from the inside.

The mechanism that produced it: resume into a live arc → it yields interesting results → interesting results justify the next experiment → repeat. Momentum substitutes for prioritization. Nothing about the arc grew more valuable; the question simply stopped being asked.

The check, cheap and mechanical: when stocking a queue, name the ROADMAP LANE each job serves (the § parallelization map table). If every job names the same lane, that is the alarm — go read the roadmap's own crux statement before adding more. Prefer stocking ACROSS lanes: the CPU-tagged lanes are explicitly disjoint and cost nothing to run alongside GPU work.

AND THE NEW MONITOR IMMEDIATELY CRIED WOLF — fix your own instrument before trusting it. lane_check.py shipped alarming on "no CPU lane running right now". Pool jobs finish in seconds-to-minutes, so that fired every cycle on a pool that had just SUCCESSFULLY completed its work — treating success as neglect, and training the reader to ignore the alert within two cycles. Then its replacement reported "never dispatched" when CPU lanes had been dispatched 15 minutes earlier, because the staleness marker did not exist yet. Two false alarms from the anti-drift tool inside ten minutes of writing it. Fixed to alarm on STALENESS (no CPU-lane dispatch in 45 min) with the marker seeded on first use. A monitor is an instrument; the rule that it must be verified before its output is trusted applies to the ones you write to enforce the rules.

QUEUED IS NOT PRIORITIZED — check POSITION, not membership (2026-07-29, same hour). After rebalancing toward the crux lane, lane_check reported crux=2 and passed. But the dispatcher is FIFO and the two gap#4 jobs sat at positions 4 and 5, behind three incumbent-lane jobs, with seven more already running — so the crux would have started dead last, hours later, while every indicator said "served". A coverage check counts membership; the schedule is what actually allocates. When rebalancing a FIFO queue, MOVE the under-served lane to the front and verify the order, or the fix is cosmetic.

COVERAGE IS NOT PROGRESS — an "unserved" lane may be a FINISHED one (2026-07-29). lane_check flagged lanes A · Affect and C · Self/Workspace as unserved. Checking the roadmap BEFORE queueing them showed both are complete: self-schema (DR-3) and false-belief (W3) are 6-seed GO, and the roadmap's own status line reads "Phase-0 keystones DR-1/DR-3/P0.3/P1.2/W3 all landed". Queueing them would have repeated, within the hour, the exact duplication the crux-lane lapse had just cost. The coverage check measures where compute goes, not whether that lane still has open work — always read the lane's roadmap STATUS before stocking it. A lane with no open work is correctly empty.

READ-THE-RECORD IS NOW ON THE EXECUTION PATH, NOT IN MEMORY (2026-07-29)

The most expensive lapse of the day was spending crux-lane GPU slots re-running a result banked five days earlier. tools/before_you_build.sh already existed to prevent exactly that — and was skipped, because running it was a thing to REMEMBER and an urgency (an owner critique about prioritization) made speed feel appropriate. Urgency defeats checks that live in memory. So the check moved onto the path the work must travel:

• tools/queue_add.sh <lane> "<cmd>" [reason] — the only sanctioned way to enqueue. It greps the findings for the runner, PRINTS every doc that already mentions it, and refuses to enqueue a runner with prior findings unless given an explicit reason, which is then recorded inline in the queue forever (#checked:on-bridge-not-rate).
• tools/lane_dispatch.sh refuses to dispatch any line lacking #checked: — it sidelines it to <queue>.unchecked (never drops it) and prints [BLOCKED]. The heartbeat alarms on a non-empty .unchecked.

Verified by unit test, not by absence of failure. The first live test looked like a pass — no leaked job — but zero lines had been dispatched (all slots busy), so the gate was never exercised. Absence of a leak is not evidence when the mechanism never ran. The gate's case-statement and sideline path were then tested directly: marked lines pass, unmarked lines block and land in .unchecked, queue intact.

pkill -f MATCHES YOUR OWN TOOLING — kill by PID (2026-07-29, third occurrence). pkill -f "[l]ane_dispatch.sh" killed the invoking shell (exit 144) AND the heartbeat monitor, because the monitor's own command text contained the string lane_dispatch.sh in its restart hint. The bracket trick only protects against the grep self-match, not against every other process whose command line mentions the pattern. Use pgrep then kill <pid>, and never embed a process name verbatim in a monitor that also watches for it (v10 splits it as 'lane''_dispatch' so the monitor can never match itself).

Also: raise SLOTS before choosing between lanes. The crux sat at the queue FRONT and still could not start, because all 7 slots were held by long-running incumbent-lane jobs — queue position does not free capacity. VRAM was 11 GB of 24 GB, i.e. ~8 slots of headroom. Raising the dispatcher 7→10 started the crux AND a second lane immediately, with no job killed and no priority call needed. Check headroom before treating a scheduling conflict as a prioritisation dilemma.

AN ALARM YOU SILENCE BY INVENTING WORK IS MIS-SPECIFIED (2026-07-29). The heartbeat alarmed QUEUE-LOW(1) while ten long-running jobs occupied every slot — a state that is entirely healthy. The first instinct was to hunt for something to enqueue, and the first candidate (the gap#5 neural reader) turned out to be a NEW BUILD rather than a runnable job: queueing it would have meant inventing work to satisfy an indicator, the same shape as the lane-monoculture failure the indicators exist to prevent.

Fixed by measuring the thing that actually matters: starvation = a short queue AND idle capacity right now (slots - running > 0). A drained queue with every slot busy is fine and no longer alarms. When an alarm's remedy is "manufacture something", the alarm is measuring the wrong quantity — fix the alarm, do not feed it.

"IS THE RUNNING JOB DOING WHAT I THINK?" — tools/device_check.sh (2026-07-29)

Every other mechanism built today is a SCHEDULING check (right work, right lane, right order, checked against the record). None asks whether a running job is doing what it appears to. The crux ran 47 minutes on the CPU with every scheduling and liveness indicator green — CPU-time tracked elapsed-time at 99%, so it was genuinely computing, on the wrong device. The runner printed the cause in line 1 (os.environ.setdefault('SIM_BACKEND','numpy') silently winning) and it was scrolled past while reading for a verdict.

bash tools/device_check.sh [--quiet] reads each running job's actual stdout via /proc/<pid>/fd/1 and reports its device; exit 1 on any CPU-bound job.

IT SHIPPED WITH A FALSE PASS AND THAT IS THE REAL LESSON. v1 pulled the log path from ps args — but the shell consumes redirects, so the path was never there, every job read unknown, and it printed "OK — no job is silently on the CPU" having determined nothing. A check that passes on no information is worse than no check, because it manufactures confidence. Fixed twice over: the path now comes from the process's real fd, and UNDETERMINED is a FAILURE, never a pass. Also handles STALE logs — a log older than its process is a leftover from a previous run (the killed CPU crux logs still said "numpy" 62 minutes later and were nearly misread as a repeat failure).

tools/engagement_check.py — the day's DOMINANT failure, made mechanical (2026-07-29)

Six instances in one session, each a different proximate cause, all the same error: a measurement returned a NUMBER while the thing being measured never happened. An unreachable threshold (a fallback branch did the work) · a soft bound that never bound · a lesion on a gate that was never declared (drift exactly +0.000000 reads as a perfect freeze) · the crux 47 min on the CPU · alpha sweeps whose control sat at ceiling · on-bridge V1 measuring orientation selectivity in neurons firing at 0.0007.

tools/lab.py already had helpers for this, but they must be imported and remembered. This runs on the ARTIFACT afterwards, so it catches probes written without them:

.venv/bin/python tools/engagement_check.py research/findings/raw/**/result.json

Two tests, both validated against a known-VOID and two known-GOOD artifacts: NEAR-ZERO ACTIVITY (a firing rate / dw / drift / sat_frac at ~0) and EXACTLY CHANCE (a score equal to 1/n — "nothing happened", not "it failed"). Control metrics (permuted, lesion, nocorpus, scramble, shuffle, abstain…) are EXEMPT: a control at zero is the control working.

A third test was BUILT AND THEN DELETED. "Identical arms → inert lever" fired on a known-GOOD artifact because, from an artifact alone, there is no way to distinguish two arms of one comparison from two different quantities that legitimately coincide (n_registered == n_rendered_exact IS the GO). A heuristic that cannot be made reliable is worse than none — it trains the reader to ignore the checker. That test belongs at probe time where the arms are known: lab.py::lever(before, after).

THE HELPERS EXIST AND I DID NOT IMPORT THEM — THREE TIMES IN ONE EVENING (2026-07-29)

tools/lab.py was written earlier the same day, explicitly to make void-arm detection EXECUTE rather than be remembered. Then three consecutive toys in one design series were written without importing it, and each shipped a different flavour of the same defect:

1. shift-invariant sums — sum(roll(x,d)) equals sum(x), so forward and reverse were equal BY CONSTRUCTION; every ratio read exactly 1.000 at all five delay settings.
2. a control that did not control — the "spreading front" travelled forward while widening, so it stayed directional and could not test the symmetry claim it existed to test.
3. a lesion divided by itself — amplification(a,Rl)/amplification(a,Rl), which returns 1.000 and cannot fail.

Two of the three were caught only by their SIGNATURE (identical values across every arm), not by foresight. The third was caught only when its number came out on the wrong side.

THE RULE, and it is about ordering, not knowledge: write the CONTROLS FIRST, with from tools.lab import lever, void_if, undefined_if_empty at the top of the file, BEFORE writing the mechanism under test. A lesion must be a genuinely DIFFERENT quantity from the arm it is compared against — if you can reach the lesion by editing one argument of the treatment expression, it is probably the treatment.

AND: an error-rate spike is a stop signal. Three implementation bugs in one evening on one series is a fact about the builder, not the design. The correct response is to STOP that line and hand it off with an honest state note — not to iterate a fourth time. Continuing produces void arms that later read as negatives and poison a mechanism that was never actually tested.

A PIPE DEFEATS && — the exit code you gate on is the LAST command's (2026-07-29, second occurrence). check_docs.py | tail -2 && git commit committed a W2 violation, because a pipeline's status is tail's, not the checker's. Earlier the same day a ; instead of && let a commit through while the checker printed FAIL. Two different ways to lose the same gate in one session. Gate on the exit code with no pipe:

if .venv/bin/python tools/check_docs.py > /tmp/dc.txt 2>&1; then git commit ...; else cat /tmp/dc.txt; fi

(Or set -o pipefail.) The lesson generalises past docs: whenever a check gates an action, verify the gate actually gates — piping the check's output for readability silently disarms it.

lr=0 IS AN ARM, NOT AN ASSUMPTION — a structural head-start reads as a learned effect (2026-07-29, TWICE in one evening)

Any claim that something was LEARNED / ACQUIRED / SELF-ORGANISED must carry an UNTRAINED (or lr=0) arm in the same run. Not "check the weights changed" — dW != 0 is satisfied by the DECAY term alone and proves nothing.

Two retractions the same evening, same class, caught by the same one-line check:

• on-bridge: "learned place tuning, selectivity 1.61-1.73x, 11-12 distinct prefs." The untrained network read 1.61-1.73x with the same distinct-pref counts, identical to two decimals per seed — it was the peak/mean of the random initialisation. The reader had never fired (w0=0.5 vs a firing threshold of 200-300 that the same session had already measured), so Hebbian's post-factor was absent and potentiation was impossible. I had "checked for a void arm" — but checked the WEIGHT BOUND, not the post-factor.
• off-substrate: "reader ACQUIRES tuning by k-WTA Hebbian, 9.1x, handed nothing" — the docstring's own words. At lr=0 it read 9.33-9.38x, i.e. ALL of it, and learning made it worse on 3/3 seeds. The function seeds each reader with a place field (W[j, seeds[j]-2:seeds[j]+3] += 0.05) before any learning.

The tell in both: selectivity tracked STRUCTURE, not training. On-bridge it moved 1.7x -> 4.5x -> 11.1x purely with connectivity density (a cell wired to fewer inputs mechanically has a higher peak/mean). If a "learned" metric responds to a WIRING parameter, suspect the wiring is the whole effect.

Do this: (1) run lr=0 as an arm and report both numbers side by side; (2) if the mechanism needs the target to FIRE, assert spikes > 0 before reading any learning metric; (3) never quote a docstring's "handed nothing" — read the initialiser. A docstring is a claim, exactly like a comment (rule 2 above).

PROVE THE METRIC CAN DETECT THE EFFECT — an invalid instrument produces confident, reproducible, meaningless numbers (2026-07-29)

Before trusting any score, show the metric is CAPABLE of measuring the claim. Three independent skeptics killed two "LEARNED" results in one workflow not by finding a confound in the mechanism but by showing the metric was blind to the property being claimed. Every number was real, reproducible to 15 decimals, and correctly lr=0-controlled. The instrument was the defect.

The case: peak/mean selectivity was used to measure PLACE tuning. It is permutation-invariant over place index — 21 weights contiguous vs scattered score an identical 4.577236; scrambling a trained cell's weights across all 60 place inputs leaves the score bit-identical to 15 dp. It also rises with weight KURTOSIS alone, so an untrained wider-jitter draw scored 5.806 ABOVE the trained 5.753. A metric indifferent to WHERE a weight sits cannot detect a spatial field. This voided a whole sub-arc of numbers, including a density series and a "mechanism X degrades the effect" negative.

Two cheap tests that would have caught it before any mechanism was built:

1. PERMUTATION TEST ON THE METRIC ITSELF. Scramble the structure the metric claims to measure (place index, time order, cell identity) and re-score. If the score is unchanged, the metric is blind to that structure — stop and pick another. Costs one line.
2. ORACLE / POSITIVE-CONTROL CEILING. Hand-construct the ideal answer (here a sigma=5 Gaussian place field) and score it. The oracle scored 14.05 where the best trained result scored 5.97 — so the trained result was nowhere near the effect it claimed, visible immediately. Also score a random redraw with zero learning: it scored +3.28, i.e. 263% of one claimed effect.

Also: match the null to the nuisance. A "sharpening-matched null" (same sharpening, no place structure) turned a +0.367 claim into +0.0058 ± 0.0074, t=0.79. And when two structure-destroying controls disagree in SIGN (shuffled +0.367 vs permuted-sweep −0.294), neither is measuring the structure.

The generalisation: lr=0 arms, engagement asserts, and bound pre-flights all check that the MECHANISM ran. None of them checks that the MEASUREMENT is meaningful. Ask both — "did it engage?" AND "could this metric have detected the effect if it were there?"

DOES THE FUNCTION DEPEND ON THE ACTUAL WEIGHTS, OR THEIR GROSS STATISTICS? — the weight-shuffle dependency control

Trigger: any claim that a skill / signature EMERGED, or DEPENDS ON the learned or structured connectivity (a recency gradient, a compositional read, a place field, a selective assembly). Before it lands, run the distribution-preserving DEPENDENCY control (Shiu & Sterne 2024's built-in falsifier: a motor neuron fired 100/100 with the real connectome, 1/100 with weights shuffled but the global weight distribution preserved). Shuffle the trained matrix preserving its value multiset, re-run the function at a FIXED seed over >=20 shuffles, and require it to COLLAPSE. If it survives, the function rode on GROSS STATISTICS (the weight histogram / the row sums), not learned structure — the exact shape of the 2026-07-28 "compositional over a localist code" / "self-organized" overclaims.

How — tools.lab.dependency_control (numpy-only; caller supplies the metric):

from tools.lab import dependency_control
dc = dependency_control(measure_fn, W, np.random.default_rng(seed), n_shuffles=32, mode="global")
assert dc["collapsed"], dc          # real > shuffled p95 AND real >= 3x shuffled mean; else it rides on statistics

mode="per_row" / "per_col" are STRONGER (they hold the row/column SUMS fixed and still destroy within-row/col structure) — use them when a skeptic can say "it's just the row sums". Two honest caveats: it tests dependence-on-STRUCTURE, not correctness; and it is INSENSITIVE when the function rides on network GEOMETRY (a feed-forward-graph order a weight shuffle leaves intact — the gap#5 replay shuffle-bar case), so pair it with a pathway LESION — if zeroing the pathway ALSO fails to collapse the effect, the weights were never load-bearing. Worked example (the gap#5 WHEN W_ctx recency pathway, per_row 2/2 decisive): research/findings/2026-08-19-weight-shuffle-dependency-control-gap5-when-Wctx.md.

DESIGNING A CONTROL: four ways they fail silently (2026-07-30, all four hit in ONE arc)

A control that returns a number is not a control that tests something. Four distinct failures in one arc, each of which produced a confident, reproducible, WRONG conclusion — and none announced itself in its output.

1. THE NULL DIFFERS IN MORE THAN ONE PROPERTY. A "sharpening-matched null" split a weight budget by a Dirichlet, which CONCENTRATES mass on random indices — while the metric (circular resultant) rewards concentration. It scored 57% of the real effect and printed "NOT place-specific" on a result that in fact passed every legitimate control. It differed from the treatment in position AND concentration, so it isolated neither. A null must differ in EXACTLY ONE property — the one under test. The fix was to permute the ACTUAL trained increments (magnitude and concentration held fixed, position shuffled).

2. THE CONTROL LOSES VALIDITY WHEN THE REGIME CHANGES. Scrambling which cells were paired was a genuine null while all cells shared one preferred position (nothing to destroy ⇒ LEARNED ≈ SCRAMBLED correctly meant "no information"). Once the population TILED the space, every pair had an intrinsic order regardless of pairing, so the same control tested nothing — and still returned a plausible number. Re-derive what a control destroys whenever the system changes; validity is a property of the REGIME, not of the control.

3. THE CONTROL IS UNDERPOWERED ON THE MECHANISM'S OWN SCALE. A "matched-preference" null still spanned 13-15 positions = 390-450 ms of separation, against an 11.5 ms detector window — ~35x too coarse. It reduced order without removing it, then read HIGHER than the treatment. A control must differ on the SCALE THE MECHANISM OPERATES ON. Compute that scale explicitly before trusting the arm.

4. A CONFOUND CAN WORK AGAINST THE EFFECT — check the direction before dismissing it. The winner-take-all lever also cut total potentiation 31%, so the raw contrast confounded order with magnitude. Matching magnitude did not shrink the result, it GREW it (separation +0.452 → +0.571). Do not assume a confound inflates; if controlling it strengthens the effect, that is evidence FOR the effect.

AND: A BAD CONTROL CAN MAKE YOU DISCARD A GOOD MECHANISM. The k-WTA arm that ultimately solved the arc's blocker was refuted early on a metric later proven blind by identity, and its winning property — "12/12 distinct winners", asserted live in its own data — sat unread in the discarded report for hours. When re-opening a rejected arm, read its METHOD, not its verdict; the setup detail that made it work (a required plasticity_gate tag, without which the gate silently no-ops) was stated explicitly in text already in hand.

Cheap standing practice: for every control, write one sentence naming what it destroys and at what scale. If that sentence cannot be written, the control is decoration.

TRACE THE ARRAY: does the lever WRITE the same object the metric READS? (2026-07-30, THREE times in one arc)

Before running an A/B, name the array the lever writes and the array the metric reads. If they are not the same object — or not coupled by a step you can point to — the arm is UNDEFINED before it starts. Three instances in one arc, each burning a full multi-seed run to produce a flat, confident, meaningless result:

lever	it wrote	the metric read	why it did nothing
Mexican-hat "inhibition" via NEGATIVE weights	cp_connections data	somatic g_i	E/I is set by cp_traits (presynaptic TYPE), not weight sign; the negative weight merely SUBTRACTED from g_e
dendritic subunit-per-block via pathway splitting	K separate pathways	cp_v_apical	cp_v_apical is allocated per NEURON, so all pathways into a cell share ONE compartment
GABA_B to terminate the apical plateau	total_input_current_pA from cp_membrane_potential_v	the APICAL plateau above v_hold	I_gabab is computed on the SOMA (bridge.py:7330) and never touches cp_v_apical

Every one had a real, allocated, non-zero parameter that measurably moved. set_pathway_weights returned 2/2 and the CSR stored −300.0 verbatim. The gate array was written on 1800/1800 steps. GABA_B changed mean|dW|. "The parameter engaged" is NOT "the mechanism reached the target." An engagement counter on the lever's own action — which this project already banks as a rule — does not catch this, because the lever genuinely acted; it acted somewhere else.

THE CHECK, one minute, before the run: grep the engine for where the lever's quantity is consumed, and where the metric's quantity is produced. Confirm a path between them you can name. If the answer is "presumably it propagates", that IS the failure — write the assertion instead (drive the lever alone and show the metric's array changes).

AND THE PAYOFF WHEN IT FAILS: all three flats became structural findings — E/I is trait-based; apical is per-neuron; GABA_B is somatic — which together justified a multi-compartment sim/ extension far better than the prior "probably needs it". A properly-diagnosed UNDEFINED is worth more than a mislabelled NEGATIVE, but only if you diagnose it; recorded as a negative, each would have wrongly closed a live capability.

LIVENESS IS NOT PROGRESS — check THROUGHPUT against the estimate, not just "is it running" (2026-07-30)

A 24.6-hour run was reported healthy ~20 times while on track to take ~14 DAYS. Every check answered "is it computing?" (cpu-time/elapsed = 99%, device=ok, process alive). None answered "will it finish?"

The evidence was in its own log 2 h before anyone looked: arm reservoir ... (81444s) = 22.6 h for ONE ARM of ONE SEED, against a job of 15 arms and a pre-registered estimate of 8-24 h TOTAL. The step-count arithmetic behind that estimate was right; the per-step cost was never validated against a real measurement.

THE CHECK, one line, the moment the first unit completes: per-unit-time x unit-count vs the estimate. If the job prints no per-unit marker, that is a defect to fix BEFORE launching, because it makes the job unmonitorable.

Do not accept "99% CPU" as reassurance. It rules out a hang; it says nothing about rate. The project already banked "a LAUNCH-BOUND run is genuinely computing but pathologically slow — kill + re-scope" — this is that rule's throughput half, which was written down and then not applied for a whole session.

---

⛔ SAME QUANTITY? — the one failure class with NO mechanical guard (2026-07-30)

Every other trap here can be caught by a tool. This one cannot, and it produced the day's worst call: I reported "BTSP is actively damaging the field, the gap#5 GO is in question" — a retraction of a validated result — from a comparison that was never like-for-like.

What happened. The headline was circ(dW): the circular resultant of the weight CHANGE, scored against a randset null (0.6705 − 0.0822 = 0.5883). I measured circ on the FINAL weights, which are dominated by the random initial structure. With lr=0 the final weights simply are the init, and a sparse random vector over 60 place indices has a high circular resultant by construction — so lr0 "beating" the learning arm was arithmetically guaranteed and carried zero information. The lr=0 control worked perfectly; I read it against the wrong baseline.

Why no check catches it. Both numbers are real, both are correctly computed, both have the same name in conversation ("circ"), and both are plausible magnitudes. Nothing is broken. check_docs, the research gate, launch_verified and the pre-commit hook are all silent, correctly.

THE RULE — before comparing ANY two numbers, state what each is OVER:

1. What is the base? final weights vs the weight change; a raw score vs a null-subtracted difference; a sum vs a mean; a ratio vs a count.
2. Is the denominator on the same footing? "0.588 = 67% of the 0.8719 oracle" divides a null-subtracted numerator by a raw ceiling. Same defect, still live. Cf. the CLAUDE.md retraction that subtracted a MEAN from a SUM and reported the difference as an improvement.
3. Where does the headline number LIVE? If it exists only as prose — hand-computed from two columns of a markdown table, as 0.588 was, appearing in no artifact anywhere — you cannot check it, and a mismatch costs a manual trace to find. Make the runner record the headline's own quantity. That single fix (adding circ_dW) turned an unfalsifiable claim into a checkable one.
4. A metric pinned at a bound is a CEILING, not a margin. Lane B read exactly 1.000 on every gate, every seed, both backends. The GO rests on the lesion/yoked/permuted differential, never on those numbers, and they must not be used to RANK variants.

Smell test: if you are about to retract a validated result on the strength of a new number, first write down the two quantities side by side with their bases. If they differ, you have found a units bug, not a finding.