| name | strop-check-loop |
| description | Loop-engineering audit of one Strop task — is the training loop CLOSED (task↔harness contract, frozen gold, signed rubric, env, forward liveness, smoke-ability)? Read-only; verdict table + LOOP-READY / NOT READY. Use when the user types /strop-check-loop or asks "这个 task 闭环了吗 / loop-ready 吗 / 能开训吗 / check the loop". Arguments (optional): task dir; `smoke` to additionally live-run a 1-item forward. |
You are auditing whether a Strop task satisfies the loop-engineering contract — i.e. whether
/strop-train can run end-to-end on it. You CHECK and REPORT; you fix nothing and launch nothing.
Parse arguments (free text; all optional)
taskDir — first path-looking token. If absent, DETECT: list tasks/*/ excluding _template;
exactly one real task → use it; several → ask the user which one (never guess).smoke — additionally run the Tier-2 live smoke (1 real forward call; see check F). Without this
flag the audit is fully read-only and free.
Ground rules
- Read-only with respect to the repo. The optional smoke writes ONLY to the scratchpad/throwaway
paths, never into
<taskDir> or .runs/.
- Never print secret values — when checking
.env, grep for variable NAMES only.
- Never dump sealed gold content — for dev/heldout you may COUNT rows and check the
gold field
exists; do not quote gold labels of sealed splits into the report.
The checklist (severities mirror design §7.7 咬合力度: BLOCK = train cannot/must not run; WARN = runs degraded)
A. 结构 (structure)
<taskDir>/task.md exists — else BLOCK.<taskDir>/rubric.md exists — else BLOCK.<taskDir>/skill.seed.md (θ₀) exists — else WARN (the workflow bootstraps via strop-init-agent).data/train/ and data/dev/ contain non-empty *.jsonl — else BLOCK. data/heldout/ non-empty
— else WARN (train runs, but no final blind eval possible). Report item counts per split.
B. task↔harness 契约 (Tier 0 static)
5. task.md has a ## Forward block whose command template uses ALL of {skill} {heads_dir}
{head_num} {in} {out} as placeholders (not hard-coded paths) — else BLOCK.
6. task.md has a ## Output schema block — else WARN (judge scores from preds+rubric, degraded).
7. task.md declares required setup (env var names / how to load ./.env), or genuinely needs none
— else WARN.
8. task.md has a smoke-safe: yes marker — else INFO: every run's Tier-2 pre-flight smoke will be
SKIPPED; if the forward looks idempotent (writes only --out, stateless API), suggest declaring it.
C. 探活 (Tier 1 liveness)
9. The executable named in ## Forward answers --help / arg-parse (NO real forward pass; loading
./.env first is allowed) — else BLOCK.
D. 尺子与 gold 纪律 (the yardstick)
10. rubric.md contains a sign-off record — grep for HUMAN-SIGNED or MACHINE-SIGNED — else
BLOCK for a real run (the ruler must be signed before it can be optimized against; a smoke train
is tolerable). Report WHICH signer you found — human or machine — never blur the two in
your output: a machine-signed rubric means the scoring WEIGHTS were auto-accepted (/strop-auto's
default path), not human-reviewed, which changes what a downstream J-Score means (agreement with
an auto-accepted ruler, not a human-verified one). Do not credit a marker that isn't actually
there — read the line's real meaning, not just a raw substring hit.
11. <taskDir>/data/GOLD_FROZEN.md exists — else BLOCK: gold is not frozen (/strop-gold +
sign-off + merge first). (Mode-A / on-the-fly judging is a designed-but-unimplemented gap —
frozen gold is currently REQUIRED.) If it exists, grep its signer: field (human or
machine; older files predating this field have neither — report "signer: unrecorded" rather
than guessing) and report which one alongside check 10's rubric signer — never present a
machine-signed gold as if it were human-verified.
12. Every row of every split carries an inline gold field (count rows with gold vs total, per
file — e.g. with grep -c, without printing contents) — mismatch → BLOCK.
13. If the task ships a vocabulary validator (e.g. validate_gold.py), run it read-only — failures
→ BLOCK.
E. 环境 (env)
14. Every env var NAME the task declares (e.g. STROP_BASE_URL / STROP_API_KEY / STROP_MODEL)
is defined in ./.env or the environment — else BLOCK. Names only, never values.
15. The forward's runtime deps import cleanly (e.g. python -c "import openai" for the reference
infer.py) — else BLOCK.
F. 冒烟 (Tier 2 — ONLY if the user passed smoke AND task.md declares smoke-safe: yes)
16. Substitute the ## Forward template with: {skill}→<taskDir>/skill.seed.md, {head_num}→0,
{in}→a 1-item slice of data/train/*.jsonl (train only — never sealed splits), {out}→a
throwaway path in the scratchpad. Run it. Verify the preds row + sidecar appear and conform to
## Output schema — failure → BLOCK (report command, exit code, stderr tail).
If smoke-safe: yes is absent, do NOT run — report SKIPPED (gated).
G. baseline-lift 可缓存性 (DERIVED — INFO/suggest only, NEVER a blocker)
This does NOT gate training — it only tells the eval layer whether the post-training baseline-lift
({bare, θ₀, θ*} on dev) can use the frame-hash cache (compute once, reuse across runs) or must
recompute each run. DERIVE it from signals already gathered — do not add new machinery:
17. baseline-cacheable = yes ⟺ the forward is deterministic/idempotent (proxy: smoke-safe: yes
is declared — a temp=0, --out-only forward) AND the judge scores against frozen gold
(data/GOLD_FROZEN.md exists + rubric signed — HUMAN-SIGNED or MACHINE-SIGNED, either makes
the rubric a fixed frame component). Both hold → the bare/θ₀ dev scores are a pure function of
the frame (student model + dev + rubric + seed + forward code + judge), so
baseline_cache.py's hash-cache is valid. Report baseline-cacheable: yes (frame-hash cache will engage).
18. If the forward is non-deterministic (no smoke-safe, temp>0, or a Mode-B agent forward) OR gold
isn't frozen / rubric unsigned → baseline-cacheable: no → the baseline recomputes each run (or is
skipped). Report baseline-cacheable: no (<which condition failed>) — baseline-lift will recompute, not cache. Still just INFO — training is unaffected.
19. --no-skill (bare arm) availability: probe the forward's --help for --no-skill. Present → the
lift is a full 3-arm {bare, θ₀, θ*}; absent → it degrades to 2-arm {θ₀, θ*}. INFO only.
Report (the deliverable)
A single markdown verdict table — one row per check: # | 检查项 | 结果 (✅/⚠️/❌/➖) | 说明 —
followed by:
- verdict line:
✅ LOOP-READY (no BLOCKs) / ❌ NOT READY — blockers: <list>;
- sign-off note (from checks 10–11, one line):
rubric: human|machine|unsigned · gold: human|machine|unrecorded — surface this even when nothing BLOCKs (e.g. a smoke-tolerant run with
an unsigned rubric); if either is machine, restate the applicable watermark line so a reader can
never mistake a machine-signed loop for a human-verified one;
- eval note (from group G, one line):
baseline-lift: cacheable yes/no · N-arm — so the user knows
whether the post-training baseline comparison caches and whether it's 2- or 3-arm. Never a blocker.
- for each blocker, the concrete fix (which file/command);
- the suggested next command: blockers → the fix; ready but never trained →
/strop-train smoke
first; smoke run already green → /strop-train.
