メニュー
Aggregate a retort run's findings.jsonl into a machine-readable assessment.json summary with severity counts, penalty score, requirement coverage, and top findings.
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
| name | file-run-issues |
| description | Aggregate a retort run's findings.jsonl into a machine-readable assessment.json summary with severity counts, penalty score, requirement coverage, and top findings. |
| type | anthropic-skill |
| version | 2.0 |
evaluate-run produces a findings.jsonl per run — one JSON object per observation. This skill aggregates those findings into assessment.json, a compact machine-readable summary used by FindingsScorer and downstream reporting. It does not create beads issues or GitHub issues.
evaluate-run used, e.g. experiment-1/runs/language=rust_model=opus_tooling=beads/rep2/info): Skip findings below this severity when computing counts. Options: critical, high, medium, low, infofalse): Print the assessment JSON without writing assessment.jsontest -f {run_dir}/findings.jsonl || { echo "no findings.jsonl — run evaluate-run first"; exit 1; }
Read the file line-by-line. Each line is one finding with at minimum these fields:
{"id": "R3", "kind": "requirement_missing", "severity": "high", "title": "...", "evidence": "...", "suggestion": "..."}
Drop findings whose severity is below min_severity. The severity ordering from highest to lowest is: critical, high, medium, low, info.
Count how many findings fall into each severity bucket:
{"critical": 0, "high": 2, "medium": 5, "low": 3, "info": 1}
start = 1.0
subtract: critical * 0.25 + high * 0.10 + medium * 0.03 + low * 0.01
clamp result to [0.0, 1.0]
A run with no findings scores 1.0. A run with one critical finding scores 0.75. A run with four critical findings scores 0.0 (clamped).
Select the top 5 findings by severity (critical first, then high, medium, low, info). Within the same severity level, preserve the original order from findings.jsonl. Include all fields from the original finding object.
Count findings with kind in requirement_missing or requirement_partial — these represent requirements the agent did not fully implement. Estimate total requirements from R<N> IDs present in findings plus any implemented ones (inferred from evaluation.md if available, otherwise estimate from the highest R-number seen).
requirement_coverage = implemented_count / total_requirements
If total requirements cannot be determined, set requirement_coverage to null.
cat {run_dir}/stack.json | jq -r '.model // .agent // "unknown"'
If stack.json is absent or has no model/agent field, use "unknown".
Write atomically (via .tmp rename):
{
"severity_counts": {"critical": 0, "high": 2, "medium": 5, "low": 3, "info": 1},
"penalty_score": 0.67,
"top_findings": [...],
"requirement_coverage": 0.75,
"model": "haiku",
"evaluated_at": "2026-04-18T21:00:00Z"
}
Constraints:
{run_dir}/assessment.json.tmp then rename to {run_dir}/assessment.json.evaluated_at MUST be an ISO 8601 UTC timestamp.penalty_score MUST be rounded to 4 decimal places.requirement_coverage MAY be null if total requirements cannot be determined.Print a terminal-readable summary:
Assessment written to {run_dir}/assessment.json
Severity counts: critical=0 high=2 medium=5 low=3 info=1
Penalty score: 0.6700 (1.0 = clean, 0.0 = critical failures)
Req coverage: 75.0%
Model: haiku
Top finding: [high] No pagination support on GET /books
If --dry-run was specified, print the JSON to stdout and skip the file write.
--dry-run by only printing what would be written.FindingsScorer in src/retort/scoring/scorers/findings.py reads {run_dir}/assessment.json and returns penalty_score directly.evaluate-run completes.assessment.json is absent, FindingsScorer returns 0.5 (neutral) rather than failing.findings.jsonl is empty
stack.json is missing
model: "unknown". Do not abort.assessment.json.tmp rename fails (permissions)
Refresh the data tables in optimal-blog.md from master.db. Checks the data for integrity problems FIRST, then runs the generator that picks per-language winners and splices every GEN-marked table, then reconciles the surrounding prose. Use after new experiment results land, or when the optimal-blog numbers are stale.
Use when working in a repository that uses bd or Beads for durable project task tracking, issue dependencies, blocker management, multi-session handoff, or shared work memory. Trigger when the user asks to find ready work, claim or close tasks, create follow-up work, inspect blockers, recover project context, or choose between local planning and persistent project tracking.
Determine the TRUE cause of a failed retort run before attributing it. Ground-truth every failure (run its tests, read its agent logs, inspect its workspace) and classify it as an infrastructure false-fail, a genuine model miss, or an environment issue — never trust the gate verdict or a log signature alone. Use whenever a run is recorded failed (test_coverage=0 / gate fail), a pass rate looks low, or you're deciding whether a "failure" is real before reporting or proceeding.
Evaluate a single retort experiment run. Score the generated code against the task's TASK.md requirements, run its build and tests, compute metrics, and emit a structured evaluation report plus a machine-readable findings file.
Compare evaluated runs in a retort experiment along factor dimensions. Surfaces effects of each factor, aggregates across replicates, and highlights cells that diverge qualitatively — complementing (not replacing) retort's ANOVA analysis.
Summarize the architecture of code generated by a single retort run. Produces module-level structure, interfaces, and control flow in a form suitable for cross-run comparison — not a full codebase-summary.