| name | update-optimal-blog |
| description | Refresh the data tables in optimal-blog.md from master.db. Checks the data for integrity problems FIRST, then runs the generator that picks per-language winners and splices every GEN-marked table, then reconciles the surrounding prose. Use after new experiment results land, or when the optimal-blog numbers are stale. |
| type | anthropic-skill |
| version | 1.0 |
Update optimal-blog.md
Overview
optimal-blog.md records what to run today, per language and task size. Its data
tables are not hand-written — they are generated from master.db by
the retort report optimal subcommand (code in
src/retort/reporting/optimal.py) and live between
<!-- GEN:<key> START/END --> markers. This skill is the safe procedure for refreshing
them: check the data before you trust it, regenerate, verify the round-trip, then fix
any prose whose numbers moved.
The order matters. master.db does not record the full stack/config (see the health
gaps below), so a blind regenerate can silently publish wrong numbers. Always run the
health check first and stop if it reports anything new.
Why per-language, not aggregates
A single cross-language reliability number is misleading — it blends a stack's strong
languages with its weak ones (local Qwen passes Python/Go but fails Rust; Opus 4.8 dips on
Java). The generator's centrepiece is the per-language success-rate matrix; the
leading-stacks routine aggregate is explicitly labelled the least-useful number. Keep it
that way — do not "promote" an aggregate back into the recommendation.
Steps
1. Health-check master.db FIRST (gate)
retort report optimal --health
Compare against the known, accepted gaps (already documented in the blog's Keeping
this current section):
- ⚠️ No sampling columns /
max_context_tokens unpopulated — the qualified config is
curated in FEATURED_STACKS, not filtered from data.
- ⚠️ ~250 rows have a blank
model (local provenance bug) — attributed by experiment slug.
- ⚠️
experiment-11, experiment-29 not ingested.
You MUST STOP and surface to the user if the report shows anything beyond those:
- "Unmapped model strings" — a new model appeared that no featured/legacy entry
covers. Decide whether it's a new featured stack (add to
FEATURED_STACKS) or legacy
(add to KNOWN_NONFEATURED) before regenerating. Publishing without deciding would drop
it silently.
- New experiment dirs not in master.db — results exist on disk but aren't ingested;
the refresh would omit them. Re-ingest first, or note the omission to the user.
- A jump in the blank-model count — the harness may have regressed; a run recording no
model is invisible to the tables. (The fix landed in
src/retort/playpen/runner.py
stack_metadata(); if new blanks appear, that path is being bypassed.)
2. Regenerate the tables
retort report optimal --write optimal-blog.md
This splices every GEN:* block (leading stacks, per-language matrix, per-language
winners, prompt method). It only touches text between markers.
3. Verify the round-trip
retort report optimal --write optimal-blog.md
git diff --stat optimal-blog.md
A second --write MUST produce no further change — the tables are idempotent. If
git diff shows table cells changed, that's the real new data; if it shows nothing, the
blog was already current.
4. Reconcile the prose
The generator owns the tables, not the sentences around them. After a refresh, read
the diff and fix any prose that quotes a number that moved:
- The bullets under Leading stacks (e.g. "Opus 4.8 ~0.59 on hard").
- The per-language recommendation table (the curated one with the prompt/testing column)
and its
†/‡ footnotes — reconcile its picks against the generated matrix and winner
table. A cell that flipped qualified↔unqualified (e.g. a language gaining local support)
changes the recommendation.
- The decision procedure list.
- The language split sentence if a language moved between local and cloud.
You MUST NOT edit numbers inside the GEN markers by hand — re-run the generator instead.
5. Report
Tell the user: which table cells changed, any prose you reconciled, and — if step 1 found
anything — what you stopped on. If nothing changed, say the blog was already current.
Constraints Summary
- You MUST run
--health and clear it against the known gaps before --write.
- You MUST NOT hand-edit between
GEN:* markers; the generator is the source of truth.
- You MUST NOT introduce or re-elevate a cross-language aggregate as a recommendation —
per-language success rates are the point.
- You MUST verify idempotency (step 3) before considering the update done.
- A new model string or un-ingested experiment is a STOP-and-ask, not a silent skip.
Troubleshooting
--health reports an unmapped model — add it to FEATURED_STACKS (with a models
list and short column name) if it should appear in the blog, or to KNOWN_NONFEATURED
if it's legacy/a serving variant. Re-run health until only the accepted gaps remain.
Blank-model count grew — the harness stopped recording model for some runs. Check
stack_metadata() is still called by every runner's provision()
(local_runner, metaharness_runner, docker_runner); a new runner or a bypass would
reintroduce the original bug. Do not paper over it in the generator.
A local language's number looks too low — confirm the curated selection in
FEATURED_STACKS still points at the tuned-config experiments, not an all-experiment
average (which includes early bad-config runs and understates the tuned reality).
