strop-eval

name	strop-eval
description	Score one or more prediction files ("arms") against a Strop task's frozen gold with the fixed judge → a J-Score comparison table (the standalone eval / transfer entry, outside the training loop). Use when the user types /strop-eval or asks to "对比 J-Score / eval these preds / transfer eval / bare vs skill 对比". Arguments: arms (label=preds path pairs — required), task dir, split=train|dev|heldout.

You are launching a Strop standalone eval — strop-eval.workflow.mjs: same judge, same rubric, same frozen gold across all arms; that constancy is the ONLY thing that makes J-Scores comparable.

Parse arguments (free text)

• arms — REQUIRED: one or more label=path/to/preds.jsonl pairs (e.g. bare=.runs/news-labeling/baselines/flash-bare.preds.jsonl theta0=...). If the user gave none, look at what they pointed to in conversation; if still ambiguous → ask, never guess. Each arm becomes { label, preds } (plus usage/cost if the user names a usage sidecar).
• taskDir — first path-looking token that is a task dir. If absent, DETECT: list tasks/*/ excluding _template; exactly one real task → use it; several → ask.
• split — which frozen-gold split to score against. Default dev for exploratory comparisons. ⚠️ split=heldout reads the blind set's gold: that is a sealed-set event — allowed for the FINAL transfer table, but confirm intent first (same spirit as /strop-train heldout).

Pre-flight (cheap, local)

1. Every arm's preds file exists and is non-empty jsonl → else STOP naming the missing arm.
2. <taskDir>/data/<split>/*.jsonl rows carry inline gold and <taskDir>/data/GOLD_FROZEN.md exists → else STOP (nothing frozen to score against; /strop-gold first).
3. <taskDir>/rubric.md exists with a sign-off record (HUMAN-SIGNED or MACHINE-SIGNED) → else STOP. Note which signer in your result summary — a machine-signed rubric means the J-Score below measures agreement against an auto-accepted ruler, not a human-verified one (see /strop-auto).

Launch

Workflow({ scriptPath: "strop-eval.workflow.mjs",
           args: { taskDir, split, arms: [{label, preds, usage?}, ...] } })

Background; tell the user what's being compared and that a completion notification will arrive.

On completion

• Failure: fail-fast — diagnose root cause from the error/transcript, report, do NOT auto-rerun.
• Success: relay the result table — arm × {J-Score, n, cost} — and read it for the user (e.g. bare vs +θ₀ vs +θ* deltas; whether "cheap model + trained instruction ≈ big model bare" held). Remind that scores are AGREEMENT with reference gold, not accuracy.