| name | strop-eval |
| description | Score one or more prediction files ("arms") against a Strop task's frozen gold with the fixed judge → a J-Score comparison table (the standalone eval / transfer entry, outside the training loop). Use when the user types /strop-eval or asks to "对比 J-Score / eval these preds / transfer eval / bare vs skill 对比". Arguments: arms (label=preds path pairs — required), task dir, split=train|dev|heldout. |
You are launching a Strop standalone eval — strop-eval.workflow.mjs: same judge, same rubric,
same frozen gold across all arms; that constancy is the ONLY thing that makes J-Scores comparable.
Parse arguments (free text)
arms — REQUIRED: one or more label=path/to/preds.jsonl pairs (e.g.
bare=.runs/news-labeling/baselines/flash-bare.preds.jsonl theta0=...). If the user gave none,
look at what they pointed to in conversation; if still ambiguous → ask, never guess. Each arm
becomes { label, preds } (plus usage/cost if the user names a usage sidecar).taskDir — first path-looking token that is a task dir. If absent, DETECT: list tasks/*/
excluding _template; exactly one real task → use it; several → ask.split — which frozen-gold split to score against. Default dev for exploratory comparisons.
⚠️ split=heldout reads the blind set's gold: that is a sealed-set event — allowed for the FINAL
transfer table, but confirm intent first (same spirit as /strop-train heldout).
Pre-flight (cheap, local)
- Every arm's preds file exists and is non-empty jsonl → else STOP naming the missing arm.
<taskDir>/data/<split>/*.jsonl rows carry inline gold and <taskDir>/data/GOLD_FROZEN.md
exists → else STOP (nothing frozen to score against; /strop-gold first).<taskDir>/rubric.md exists with a sign-off record (HUMAN-SIGNED or MACHINE-SIGNED) → else
STOP. Note which signer in your result summary — a machine-signed rubric means the J-Score below
measures agreement against an auto-accepted ruler, not a human-verified one (see /strop-auto).
Launch
Workflow({ scriptPath: "strop-eval.workflow.mjs",
args: { taskDir, split, arms: [{label, preds, usage?}, ...] } })
Background; tell the user what's being compared and that a completion notification will arrive.
On completion
- Failure: fail-fast — diagnose root cause from the error/transcript, report, do NOT auto-rerun.
- Success: relay the result table —
arm × {J-Score, n, cost} — and read it for the user
(e.g. bare vs +θ₀ vs +θ* deltas; whether "cheap model + trained instruction ≈ big model bare"
held). Remind that scores are AGREEMENT with reference gold, not accuracy.
