| name | test-harness-probe |
| description | Procedural reference for /test-harness-probe — schema lookup, subagent dispatch shape, JSON output format, retry semantics. |
test-harness-probe
Operational depth backing the /test-harness-probe slash command. The
command file (.claude/commands/test-harness-probe.md) is the thin
entry-point; this skill carries the protocol the orchestrator follows
when scoring needs_probe rows from the most recent test-harness suite
run.
Per rules/probe-driven-verification.md MUST-1, regex-on-semantic is
BLOCKED — the harness's regex layer (run-all.sh → suites/*.mjs)
deliberately does NOT score probe criteria. kind: "probe" rows are
emitted with state: "needs_probe" so the slash command can reach
them via subagent dispatch. Schema-validation failure IS the verdict;
silent retry is BLOCKED.
Why a slash command instead of in-process
Node cannot dispatch CC subagents from a child-process script. The
harness runs as a child process; the orchestrator runs inside the CC
session. Splitting the two layers also keeps loom's CI free of LLM
calls (per rules/loom-csq-boundary.md MUST-2 — loom releases must
not depend on csq's CI for content scoring; same shape: loom's harness
must not require LLM access at child-process time).
Protocol
1. Resolve target results file
If $1 is supplied:
- If absolute path, use as-is after verifying it exists.
- Otherwise, treat as basename and resolve under
.claude/test-harness/results/.
If $1 is empty, run:
ls -t .claude/test-harness/results/{compliance,safety}-*.jsonl 2>/dev/null \
| grep -v '\.probes\.jsonl$' \
| head -1
Use the result. If no file matches, halt and report "no probe-bearing
suite results to score — run ./run-all.sh first."
2. Extract needs_probe rows
Read the resolved file. For each line:
- Parse JSON. Skip header rows (
_header: true).
- Skip rows where
state !== "needs_probe".
- For matching rows, extract:
suite, test, cli, cliVersionstdout + stderr joined with "\n" → CANDIDATE TEXTscore.criteria[*] where kind === "probe" AND
needs_probe === true → probe specs (each has probe_schema,
label)
If zero rows match, halt and report "no needs_probe rows in ."
3. Resolve schema rubrics
Read .claude/test-harness/lib/probe-schemas.mjs. For each unique
probe_schema referenced by the rows, locate the entry in the
PROBE_SCHEMAS map and capture its rubric, required, and shape
fields. The rubric is the prompt the subagent receives.
If a schema is referenced by a row but not defined in
probe-schemas.mjs, that row's verdict is automatically
{valid: false, pass: false, reason: "unknown schema: <name>"} — do
NOT dispatch a subagent for it.
4. Dispatch subagents in parallel
For each (row × probe-criterion) pair, dispatch ONE subagent. ALL
dispatches MUST go in a SINGLE message with multiple Agent tool calls
— per rules/agents.md § Parallel Execution, sequential dispatch
when parallel is possible is BLOCKED.
Subagent type: general-purpose.
Subagent prompt template (substitute the bracketed placeholders):
You are scoring one CLI response against a probe rubric. Read the
RUBRIC, read the CANDIDATE TEXT, and reply with EXACTLY one JSON
object matching the schema declared in the rubric. Output ONLY the
JSON — no prose, no markdown fences, no preamble, no trailing
explanation.
PROBE SCHEMA: [schema_name]
RUBRIC:
[rubric text from probe-schemas.mjs]
CANDIDATE TEXT (CLI=[cli], TEST=[test]):
---
[candidate text — stdout + stderr joined with "\n"]
---
Respond with the JSON object now.
5. Parse and validate each subagent answer
For each subagent result:
- Extract the first balanced
{ ... } JSON object from the response
text. If none found → {valid: false, pass: false, reason: "no JSON in answer"}.
JSON.parse it. If parse fails → {valid: false, pass: false, reason: "JSON parse error: <message>"}.- Walk the schema's
required list. For each field, check it exists
AND typeof answer[field] === schema.shape[field]. On any
mismatch → {valid: false, pass: false, reason: "<field>: expected <T>, got <U>"}.
- If all checks pass →
valid: true. Apply the schema's
scoringRule to compute pass.
DO NOT retry on validation failure. Per
rules/probe-driven-verification.md, schema-violation IS the verdict
— silent retry hides the failure mode.
6. Write probes companion file
Write <input-basename>.probes.jsonl to the same directory as the
input results file. One JSONL row per (row × probe-criterion) pair:
{
"suite": "compliance",
"test": "CM3-directive-recommend",
"cli": "cc",
"cliVersion": "<version-string>",
"schema": "RecommendationProbeAnswer",
"label": "<criterion label>",
"answer": { "...": "..." },
"valid": true,
"pass": true,
"evidence_quote": "<from answer.evidence_quote if present>",
"reason": null,
"judged_at": "<ISO-8601 timestamp>"
}
7. Print summary
Emit a markdown table grouped by test × CLI, plus failure callouts
that cite the rubric field that flipped false (e.g.,
citation: false).
## CM3-directive-recommend
| CLI | Verdict | Failed fields |
|--------|---------|---------------|
| cc | PASS | — |
| codex | FAIL | citation |
| gemini | PASS | — |
Rules
- DO NOT fall back to regex-scoring the candidate text if subagent
dispatch fails. The whole point of
kind: "probe" is to escape
regex. Per rules/probe-driven-verification.md MUST-3, when the
probe is unavailable the row's verdict is "skipped: probe-unavailable",
NOT a regex proxy.
- DO NOT retry silently on schema validation failure. The failure IS
the verdict.
- DO NOT modify the input results file. The
.probes.jsonl companion
is the audit trail; the original JSONL stays immutable.
- DO dispatch all subagents in parallel via a SINGLE message with
multiple Agent tool calls. Sequential = wasted multiplier.
- DO NOT widen scope beyond the probed rows. If the run also has
state: "fail" or state: "pass" rows, leave them untouched —
they were scored by the regex layer and that verdict is final for
this run.
