strop-check-loop

name

strop-check-loop

description

Loop-engineering audit of one Strop task — is the training loop CLOSED (task↔harness contract, frozen gold, signed rubric, env, forward liveness, smoke-ability)? Read-only; verdict table + LOOP-READY / NOT READY. Use when the user types /strop-check-loop or asks "这个 task 闭环了吗 / loop-ready 吗 / 能开训吗 / check the loop". Arguments (optional): task dir; `smoke` to additionally live-run a 1-item forward.

You are auditing whether a Strop task satisfies the loop-engineering contract — i.e. whether /strop-train can run end-to-end on it. You CHECK and REPORT; you fix nothing and launch nothing.

Parse arguments (free text; all optional)

• taskDir — first path-looking token. If absent, DETECT: list tasks/*/ excluding _template; exactly one real task → use it; several → ask the user which one (never guess).
• smoke — additionally run the Tier-2 live smoke (1 real forward call; see check F). Without this flag the audit is fully read-only and free.

Ground rules

• Read-only with respect to the repo. The optional smoke writes ONLY to the scratchpad/throwaway paths, never into <taskDir> or .runs/.
• Never print secret values — when checking .env, grep for variable NAMES only.
• Never dump sealed gold content — for dev/heldout you may COUNT rows and check the gold field exists; do not quote gold labels of sealed splits into the report.

The checklist (severities mirror design §7.7 咬合力度: BLOCK = train cannot/must not run; WARN = runs degraded)

A. 结构 (structure)

1. <taskDir>/task.md exists — else BLOCK.
2. <taskDir>/rubric.md exists — else BLOCK.
3. <taskDir>/skill.seed.md (θ₀) exists — else WARN (the workflow bootstraps via strop-init-agent).
4. data/train/ and data/dev/ contain non-empty *.jsonl — else BLOCK. data/heldout/ non-empty — else WARN (train runs, but no final blind eval possible). Report item counts per split.

B. task↔harness 契约 (Tier 0 static) 5. task.md has a ## Forward block whose command template uses ALL of {skill} {heads_dir} {head_num} {in} {out} as placeholders (not hard-coded paths) — else BLOCK. 6. task.md has a ## Output schema block — else WARN (judge scores from preds+rubric, degraded). 7. task.md declares required setup (env var names / how to load ./.env), or genuinely needs none — else WARN. 8. task.md has a smoke-safe: yes marker — else INFO: every run's Tier-2 pre-flight smoke will be SKIPPED; if the forward looks idempotent (writes only --out, stateless API), suggest declaring it.

C. 探活 (Tier 1 liveness) 9. The executable named in ## Forward answers --help / arg-parse (NO real forward pass; loading ./.env first is allowed) — else BLOCK.

D. 尺子与 gold 纪律 (the yardstick) 10. rubric.md contains a sign-off record — grep for HUMAN-SIGNED or MACHINE-SIGNED — else BLOCK for a real run (the ruler must be signed before it can be optimized against; a smoke train is tolerable). Report WHICH signer you found — human or machine — never blur the two in your output: a machine-signed rubric means the scoring WEIGHTS were auto-accepted (/strop-auto's default path), not human-reviewed, which changes what a downstream J-Score means (agreement with an auto-accepted ruler, not a human-verified one). Do not credit a marker that isn't actually there — read the line's real meaning, not just a raw substring hit. 11. <taskDir>/data/GOLD_FROZEN.md exists — else BLOCK: gold is not frozen (/strop-gold + sign-off + merge first). (Mode-A / on-the-fly judging is a designed-but-unimplemented gap — frozen gold is currently REQUIRED.) If it exists, grep its signer: field (human or machine; older files predating this field have neither — report "signer: unrecorded" rather than guessing) and report which one alongside check 10's rubric signer — never present a machine-signed gold as if it were human-verified. 12. Every row of every split carries an inline gold field (count rows with gold vs total, per file — e.g. with grep -c, without printing contents) — mismatch → BLOCK. 13. If the task ships a vocabulary validator (e.g. validate_gold.py), run it read-only — failures → BLOCK.

E. 环境 (env) 14. Every env var NAME the task declares (e.g. STROP_BASE_URL / STROP_API_KEY / STROP_MODEL) is defined in ./.env or the environment — else BLOCK. Names only, never values. 15. The forward's runtime deps import cleanly (e.g. python -c "import openai" for the reference infer.py) — else BLOCK.

F. 冒烟 (Tier 2 — ONLY if the user passed smoke AND task.md declares smoke-safe: yes) 16. Substitute the ## Forward template with: {skill}→<taskDir>/skill.seed.md, {head_num}→0, {in}→a 1-item slice of data/train/*.jsonl (train only — never sealed splits), {out}→a throwaway path in the scratchpad. Run it. Verify the preds row + sidecar appear and conform to ## Output schema — failure → BLOCK (report command, exit code, stderr tail). If smoke-safe: yes is absent, do NOT run — report SKIPPED (gated).

G. baseline-lift 可缓存性 (DERIVED — INFO/suggest only, NEVER a blocker) This does NOT gate training — it only tells the eval layer whether the post-training baseline-lift ({bare, θ₀, θ*} on dev) can use the frame-hash cache (compute once, reuse across runs) or must recompute each run. DERIVE it from signals already gathered — do not add new machinery: 17. baseline-cacheable = yes ⟺ the forward is deterministic/idempotent (proxy: smoke-safe: yes is declared — a temp=0, --out-only forward) AND the judge scores against frozen gold (data/GOLD_FROZEN.md exists + rubric signed — HUMAN-SIGNED or MACHINE-SIGNED, either makes the rubric a fixed frame component). Both hold → the bare/θ₀ dev scores are a pure function of the frame (student model + dev + rubric + seed + forward code + judge), so baseline_cache.py's hash-cache is valid. Report baseline-cacheable: yes (frame-hash cache will engage). 18. If the forward is non-deterministic (no smoke-safe, temp>0, or a Mode-B agent forward) OR gold isn't frozen / rubric unsigned → baseline-cacheable: no → the baseline recomputes each run (or is skipped). Report baseline-cacheable: no (<which condition failed>) — baseline-lift will recompute, not cache. Still just INFO — training is unaffected. 19. --no-skill (bare arm) availability: probe the forward's --help for --no-skill. Present → the lift is a full 3-arm {bare, θ₀, θ*}; absent → it degrades to 2-arm {θ₀, θ*}. INFO only.

Report (the deliverable)

A single markdown verdict table — one row per check: # | 检查项 | 结果 (✅/⚠️/❌/➖) | 说明 — followed by:

• verdict line: ✅ LOOP-READY (no BLOCKs) / ❌ NOT READY — blockers: <list>;
• sign-off note (from checks 10–11, one line): rubric: human|machine|unsigned · gold: human|machine|unrecorded — surface this even when nothing BLOCKs (e.g. a smoke-tolerant run with an unsigned rubric); if either is machine, restate the applicable watermark line so a reader can never mistake a machine-signed loop for a human-verified one;
• eval note (from group G, one line): baseline-lift: cacheable yes/no · N-arm — so the user knows whether the post-training baseline comparison caches and whether it's 2- or 3-arm. Never a blocker.
• for each blocker, the concrete fix (which file/command);
• the suggested next command: blockers → the fix; ready but never trained → /strop-train smoke first; smoke run already green → /strop-train.