model-benchmarks-refresh

# Refresh the model benchmarks page Use this skill for any refresh of `/model-benchmarks`, its source posts, model notes, or tiers. This is a curated decision-support page, not an automatic leaderboard. Read this entire file before starting. Announce that you are using it. ## Read the current state first Read `AGENTS.md`, `src/model-benchmarks.json`, `src/modelBenchmarks.ts`, `src/ModelBenchmarksPage.tsx`, `src/model-benchmarks.css`, and their tests. Check the Git diff and preserve unrelated work. The JSON is the versioned source of truth for this page. Do not fetch recommendations, rank models, shuffle sources, or discover providers at page load or runtime startup. The canonical URL is `https://happy.engineering/model-benchmarks`. The Happy picker footer says **Latest benchmarks**. The “trust me bro” title identifies this as an editorial synthesis, not an independently reproduced lab benchmark. Keep methodology in this pipeline, not a long page preamble. The checked-in [source data and model ordering](../../../src/model-benchmarks.json) contain the full claims, limitations, verification provenance and rationale. The [selection logic](../../../src/modelBenchmarks.ts) and [page component](../../../src/ModelBenchmarksPage.tsx) consume this fixed JSON. ## Principle: stability Readers come back to this page to decide what to use. Every reorder, tier move, or swapped post costs them trust and attention, so the table moves only when the evidence does. This principle governs every other rule in this file. - **A release is not a result.** A new model, a launch thread, or one hype post never moves the table by itself. New models enter where converging evidence puts them, after the incumbents of that tier, and never push an incumbent down on launch-week claims. - **Change needs converging, verified, exact-version evidence and a written reason.** Move a tier, reorder a tier, or replace a post only when independent sources on the exact versions agree, and record why in the model `rationale` and the handoff. Mixed or provider-only evidence leaves the incumbent where it is. - **Keep reviewed posts.** Replace one only when the new post materially improves exact-version relevance, methodology, coverage, verification, or recency. A newer date alone is not enough. Never replace every post in a refresh; record the old/new URL and the reason for each replacement. - **Keep array order.** Preserve existing model and source order when conclusions have not changed, and place new sources deliberately. The page sorts the wall by date (see the posts wall below); the JSON order stays editorial. Never sort the JSON by live dates, popularity, API arrival order, or search ranking. Same-tier order is a deterministic presentation choice, not an invented numeric quality gap. - **No change, no diff.** A no-material-change review produces no catalog diff, no new revision, and no commit. Do not bump the public date because you ran a search; verification logs belong in scratch. On a material change, increment `revision`, set honest UTC `updatedAt`/`evidenceCutoff`, and keep each original `publishedAt`. Correcting a recorded fact about a selected post (a wrong timestamp, a renamed handle) is material. - **Stable does not mean stale.** When a challenger tries to take the top spot and fails, say so with fresh evidence (see the lead refresher below) rather than leaving the leader's launch-week posts to argue alone. ## Shape of the chart The chart answers one question: which model should I use? It is short, and every row on it carries a recommendation. - **Four tiers: S, A, B, F.** There is no C, D or Unranked tier. A tier with no models is not rendered; never pad a tier to fill it. - **S is aggressive: one to three models in total, across both lanes.** S holds only the clear best. A model that trails an S model in most reviewed head-to-heads belongs in A, however close it is. Do not add a second Frontier model out of politeness to its lab. The cap counts across both lanes, not per lane: a per-lane cap would allow six S models on a seven-model chart, which defeats the point of S. - **S splits into two lanes.** **Frontier** holds the most capable models for hard work, ranked on quality alone. **Chores** holds models that are very good and cheap: the best value for routine work, judged on quality per cost from sources that measure both (cost per task, not list price per token). A Chores model must be clearly good, not merely cheap, and must beat the other cheap candidates on converging evidence. Either lane may hold one model; Frontier is never empty. Only S has lanes; A, B and F are single cells. - **A** is strong and worth using for some work, but beaten by S on the reviewed evidence. **B** is usable with clear limits. **F** is a major lab's current frontier model that we recommend against, because cheaper shown models beat it on converging independent evidence. F is a judgment like any other tier and needs the same evidence. - **At most seven models.** Fill the chart in this order: the S models; then each major lab's best remaining model with enough evidence; then any remaining slots to the strongest other models on converging evidence. Same-tier order follows the evidence; a spare-slot model is not placed above a stronger model merely because it entered first. - **Thin evidence means not shown.** A model without enough exact-version, independent evidence to place stays off the page. Record it in `notShown` with `reason: "thin-evidence"` and a note on what is missing. - **Every reviewed model that is not shown is recorded.** `notShown` lists each one with a `reason` (`superseded`, `outranked`, `over-limit` or `thin-evidence`), `by` where it applies, and a one-line evidenced `note`. It is provenance only and never rendered. Keep its entries until the model stops being relevant. ### Supersession When a newer generation of the same line is better, the older one leaves the page (Opus 5.5 removes Opus 5; GPT-6.1 Sol removes GPT-6 Sol). Record it in `notShown` with `reason: "superseded"` and `by` set to its successor. Recheck supersession whenever a lab releases a new generation. The one exception: keep the previous generation only when the new one draws widespread, evidenced complaints from many independent sources showing that people prefer the older one. One evaluator's complaint, a single benchmark regression, or a launch-week thread is not enough. When the exception applies, write the reason and its sources in both models' rationales. ### Coverage and identities The page covers the frontier of every major lab, not only the models in Happy's picker. Each refresh checks the latest release of each lab in `MAJOR_LABS` (Anthropic, OpenAI, Google, xAI, Meta, DeepSeek) and of other top labs (for example Alibaba Qwen, Moonshot Kimi, Xiaomi MiMo, Zhipu GLM, Mistral). Each major lab's latest frontier model appears either on the chart or in `notShown` with a reason. A model limited to a lab's own product is eligible; say so in its notes. Model IDs are `<lab>/<model-slug>`: the lab's lowercase short name and the provider's exact model name in lowercase, spaces replaced by hyphens, version dots kept (`google/gemini-4-argon`, `meta/muse-wonder-full`, `deepseek/deepseek-v4.1-flash-1017`). Models in Happy's catalog keep their Happy IDs. Once published, an ID never changes; a new build or snapshot is a new ID. ## Research 1. Prefer a Grok research subagent using an available funded provider. Ask for original X post URLs, exact quotations, dates, versions, tasks, effort, harness, limitations, and conflicting results. Read its findings yourself; discovery is not verification. 2. If the configured API-key provider is not yet available to subagents, report that limitation. A directly authorized xAI `/responses` call with `tools: [{"type":"x_search"}]` is an acceptable explicit fallback. Do not restart the daemon while work is running or repeatedly retry an exhausted provider. If no Grok path works (for example `402 usage balance exhausted`), say so in the handoff and discover candidates with web search restricted to `x.com` and the recent timelines of the evaluators below; everything after discovery is unchanged. 3. Never print keys, put them in arguments, log request headers, commit credentials, or include them in research snapshots. Use the configured provider credential privately. Save public raw answers and citations only in gitignored `.context/research/`. 4. Use only original X posts as ranking evidence. General web search can help locate a post, but snippets, blogs, release documentation, videos, and provider model listings do not establish a ranking. Official documentation may disambiguate identifiers, not supply ranking claims. 5. Look beyond X for leads. Each run, check every lab's latest release (its X account, blog or changelog) and the current boards of the main evaluators (Artificial Analysis, Arena, Epoch, Theo, FrontierCode). Use what you find to locate the original X post that carries the result. A release you find only off X still counts for coverage: record it in `notShown` until X evidence places it. ## Editorial gate Prioritize, in order: 1. Official provider release threads with substantive benchmark detail. Always record these as `kind: "provider-reported"` in the JSON, including developer accounts. 2. Established evaluators with a meaningful audience and substantive firsthand evaluation, including Theo when relevant. 3. Serious original benchmark/comparison researchers, even with a smaller audience, when their methodology is credible. Agents on Rails (@rails) is not a source for this page: readers do not weigh a Rails-only benchmark, so never cite it in a card point or the wall. It may still point you to a model worth checking elsewhere. Popularity is an eligibility signal, never proof or a vote weight. Exclude random unsupported comments, engagement bait, vague praise, recycled announcements, search summaries, stale or misidentified versions, and unsupported tier inference. Do not fill a quota with weak material. Independently open each proposed post using X's oEmbed endpoint and/or the original X page. Verify the author, post ID, publication date, and every specific claim used. oEmbed may truncate long posts: inspect the original full post before using text beyond the truncation. When x.com will not render without a browser, a read-only mirror of the same post ID (such as `api.fxtwitter.com/status/<id>`) can supply the full text and original `pbs.twimg.com` media; it is a reading aid, not a source. Derive `publishedAt` from the post ID's snowflake timestamp, `(id >> 22) + 1288834974657` ms. A link's existence is not enough. Store the verification method and limitations in the source record. If a chart image carries the evidence, inspect the actual image, transcribe its exact version labels and values, and review them. OCR/Grok interpretation alone is not verification. Never infer an intra-tier order from left-to-right chart placement unless the author explicitly specifies one. ## Exact identities and claims - Map every claim to an exact catalog model ID. Do not use family selectors or transfer a predecessor's reputation to the current family head. - Preserve task, effort, harness, benchmark revision, date, limitations, and conflicts. A mixed-model fallback is not a clean single-model result. - A source can support several models only when it explicitly covers each one. Its `modelIds` must match the models actually citing it. - `comparisonModelIds` means an actual head-to-head among those exact models, not separate mentions, a family comparison, or a comparison against an unspecified model. - Compare like-for-like before interpreting scores. Do not pool incompatible benchmark versions or resolve contradictions by averaging them away. - Separate task quality from price, token usage, vendor availability, and general intelligence metrics. Prices mentioned by sources are contextual claims, not current pricing data or tier weights. - Each model card carries a headline and two or three concise, supported points; the last point is the caveat. Every point is one claim from one post: `{ "text", "sourceId" }`, where `sourceId` is one of the model's own `sourceIds`, so the card links the claim to that post. Write a point to fit two card lines (at most 72 characters; measure the lines in a browser) and don't split a claim across posts. A comparison names the effort when it is not the model's best ("Same index: GPT-6.1 Sol at medium effort scores 48 for $0.21 a task."). - **Every number names its metric and its scale; the source is the link.** Write for a reader who has never heard of the source: say what was measured and out of what, in plain words, inside the two lines. The ↗ already points to the post, so don't lead with the publisher: "Coding-agent index: 59, to Opus 5.5's 62 and GPT-6 Astra's 55.", not "Artificial Analysis coding-agent index: …". Name a person only where they are the point, and briefly ("Theo's blind code audit: 47 of its 92 claims were false."). Never write "AA". Name each benchmark with what it tests ("FrontierCode, a test of merge-ready code", "Terminal-Bench shell test", "Code Arena WebDev, users' votes on AI-built web apps"), and give an index score a comparison that shows its scale. A fraction says what it counts. If a claim cannot be made clear in two lines, drop it or pick a clearer one. The same rule covers the headline and the badge; the legend names the sources once. - **The headline is the model's rank on one Agent Arena board, the latest, for every card.** Arena's score is "net improvement vs baseline": how much better real users' agent sessions go with the model than with the average model on the board, combining Confirmed Success, Praise vs Complaint, Steerability, Bash Reliability and Tool Hallucination ([signal definitions](https://arena.ai/resources/agent-arena-signals)). It is not a win rate or a solve rate, and a raw `+20.9%` means nothing to a reader, so the card never shows it alone. `arenaBoard` names the board's post and its #1 and #2 (name, effort, score), shown on the page or not; each model's `arena` is `{ rank, effort, score }` as that post labels it. The page derives the second line: #1 shows its lead over #2 ("6.2 points ahead of #2 GPT-6 Astra (Max) in real users' agent sessions"), every other model its gap to #1 ("7.4 points behind #1 Opus 5.5 (Max) in real users' agent sessions"). Gaps are percentage points, as Arena quotes them, never percent. Arena lists one entry per model at its labeled effort; record that effort exactly (`Max`, `High`, `xHigh`), and if a board lists several efforts for one model, use its best-scoring entry. Every model with an `arena` entry cites the board post in its `sourceIds`; when a new board comes out, move every card to it at once (the board replaces a model's weakest post if it is at the four-post cap) and rewrite the board post's `summary` as `#<rank> <name> (<effort>) +<score>%` entries, which the tests check each entry against. A shown model not on the board yet has no `arena` and its card says "Not on Agent Arena yet". The one-line legend under the table explains the metric and links the board post; keep it. - **Every coding-agent index bullet quotes one version of AA's Coding Agent Index, from one post.** Its versions don't compare (v2.0 scores ran 7 points below v1.5 on average and reordered the top), so when AA publishes a new version, move every such bullet to it at once, swap the post in for the old one (on the wall too), and rewrite any bullet whose claim changed. Use the chart's whole-number bar labels so every model reads at the same precision. A test checks that these bullets share one source. - **The page shows no prices.** Kirill's decision, Oct 23: API prices mislead the many readers who pay through a $200 subscription (Claude Max, ChatGPT Pro), and no source publishes a subscription-adjusted cost per task, so cards carry no price chip, the legend has no price line, and the JSON has no `price` or `pricing`. Don't add them back on a refresh. A point may still compare cost when the comparison justifies a tier, such as GPT-6.1 Sol's value case for Chores ("FrontierCode coding test: 49.1%, at under a tenth of Opus 5.5's cost."); it must read on its own. Background, checked Oct 23: Arena's Agent Arena costs (p25/p50/p95 median cost per task) and AA's indexes (AA: "Costs are pay-per-token API costs, not subscription prices") are API costs; aiandtractors.com publishes "Estimated subscription / task" beside AA's API cost per attempt but marks Opus 5.5, GPT-6.1 Sol, Gemini 4 Argon, Grok 4.7 and Muse "Unknown" (only GPT-6 Astra, $0.213, a September proxy); FinOps LLM and TokenPost (SemiAnalysis) publish plan values in API dollars, not per-task figures. Revisit only if Kirill asks and a source publishes subscription-adjusted prices for every shown model. - Each model records `releasedAt` and `releaseUrl`: the lab's own X post announcing general availability (a public API or product), not a limited preview, with `releasedAt` taken from that post's snowflake. Gemini 4 Argon dates from its global availability on Oct 13, not its Sep 30 trusted-tester launch. A new model's card shows a small badge in its top-right corner with Lucide's `baby` icon and its age in whole days (`4d`) while it is `NEW_MODEL_DAYS` (14) days old or less. The page computes the age in the browser at page load, so the badge ages and disappears without a redeploy; never hard-code an age or a "new" flag. - Thin evidence means the model is not shown (see Shape of the chart). Do not invent a complete ladder. ## Four-post cap - Hard maximum: **four selected posts per exact model**. Prefer one meaningful official post plus up to three strong independent evaluations, but fewer is fine. - Every selected post in a thread counts. A thread is not unlimited embeds. A shared post counts against every model it supports and appears on the wall at most once. - Research candidates separately from the selected public source list; replacements follow the stability principle above. - Source age below the embed refers to publication, never last verification. Keep the absolute date accessible. ## Lead refresher The first S Frontier model is the leader. When a credible challenger launches and claims the top spot, treat the outcome as a material change either way. The `leadCheck` on the leader records the outcome: 1. Within a few days of the launch, research exact-version head-to-heads of the leader against the challenger: independent evaluations first, provider charts as context. 2. **Leader held** (the evidence is mixed, provider-only, or favors the leader): keep it first. Add one fresh, verified head-to-head post to its `sourceIds`. At the four-post cap, replace its weakest or most superseded post and record why. Set `leadCheck`: - `reconfirmedAt`: the UTC date of the review. Never in the future, never before a cited post. - `heldAgainst`: the challengers' exact model IDs. Each must be shown on this page. - `sourceIds`: posts from the leader's own list that compare the leader with the challengers. Between them they must cover every `heldAgainst` model in `comparisonModelIds`. - `note`: the first bullet on the leader's card, at most 52 characters, e.g. "Still on top after GPT-6.1 Sol, Sonnet 5.5." - `reason`: the provenance. Name the evidence on both sides, including where the challenger won. 3. **Challenger took the lead** (converging independent, exact-version evidence): move it first, explain the move in both rationales, and drop the old leader's `leadCheck`. Do not move a challenger on its own launch chart or one evaluator's run. 4. A later challenger replaces `heldAgainst` and the date. Keep earlier challengers only while their head-to-head stays in the leader's posts. Remove a `leadCheck` once it no longer describes a recent contest; it must not become a permanent badge. The page shows this only as the first bullet on the leader's card: the `note`, linked to the first of the check's `sourceIds`, then "Rechecked Oct 21." Keep the `note` to 52 characters ("Still on top after Muse Wonder Full, Gemini 4 Argon.") so the bullet fits two lines, and keep the card to three bullets in total, so a leader with a `leadCheck` has two `points`. No banners, badges on other rows, or prose. ## Deleted posts, privacy, and shared policy Embed failure alone does not prove deletion. Confirm availability separately. Remove confirmed private/deleted reproduced content from the next reviewed revision as appropriate; explain the removal and reevaluate dependent placements. Keep links and clearly labeled fallback summaries only when appropriate. Never secretly rerank an already published revision because X is unavailable. The user explicitly chose automatic official X embeds. Load the shared widget automatically, render posts as they approach the viewport, and use `dnt: true`, `conversation: 'none'`, with media retained. This contacts X; do not claim the page is third-party-request-free. Never inject untrusted oEmbed HTML or weaken CSP broadly. Keep a readable author/summary/source-link card only as the blocked/timeout fallback. ## Keep the page minimal The visible page is a centered title with the tiny updated date and GitHub **pipeline ↗** link side by side beneath it, a centered tier table of model cards with a short legend under it (the Agent Arena headline and the two Artificial Analysis indexes), and a wall of at most eight original embedded posts. There is no filter: cards are not buttons, and their links go straight to the posts. No hover popovers or tooltips, "+" toggles, selection, tier headings over the posts, head-to-head toggle, provider-reported label, consent gate, “How we curate” section, curator essays, “Happy’s read”, source-scope panels, not-shown list, or ceremonial preamble. Empty tiers are not rendered. - **Cards.** Every card shows the model name, its Agent Arena headline (rank, effort and the gap line, linked to the board post), and its two or three points, each linked to its post. All cards share one width and height (`--card-width`, 244px, and 236px in `src/model-benchmarks.css`), about twice the old name chip, sized for the headline and three two-line points; if new copy needs more room, shorten it before growing the card. Points may wrap to two lines, never three. The new-model badge reads "9 days old". - **Claim links.** No underline at rest, a thin underline on hover, and a small grey ↗ raised at the top right of the link's last word (orange on hover), never wrapped onto a line of its own. - **S row.** Every row of the tier table is the tier letter and one cell, so the letter column has one width. S's cell holds a nested table: one row, a column per lane, each headed by a horizontal FRONTIER or CHORES label (reddish on a pale S tint, a 1px rule under it) with its cards below, and a 1px rule between the columns, like the main table's. The S cards sit on the card grid of the rows below: Opus 5.5 directly over A's first card and the Chores column starting on A's second card column. That holds by construction: cards in every row are `--card-gap` apart (a cell's padding, the 1px rule and the next cell's padding), and the Frontier column is `--cards` card columns wide; a test pins those rules. Check it on every layout change by measuring the left edge of each S and A card in a browser (they match to the pixel) and the tier letter cells (one width). With one lane, its column spans the row. On phones each row becomes the tier letter beside stacked cards, Frontier's label and cards above Chores', at full width. - **Posts wall.** `wall` lists at most eight post IDs, each once, picked in order of value to S: first the leader's lead check and the leader against #2, then the Chores pick's value, then the rest, so that every shown model has at least one post on the wall. The page shows them newest first by `publishedAt`, with no grouping or headings. Card points may link to posts that are not on the wall. Pick posts that stand alone: a quote tweet embeds the quoted post as a second block and a reply can drag in its thread, so record `quotes` and `replyTo` on sources and keep both off the wall. X's `cards: 'hidden'` option does not help: it hides the post's chart and keeps the quoted post. Arena's Oct 20 Agent Arena board quotes its Code Arena post, so it stays a card link and not a wall post; Theo's audit and FrontierCode carry the lead on the wall. - **Wall layout.** Two columns of 330–350px embeds on desktop, one on phones. Never split a post across columns: the list is a CSS grid of 4px rows in which each post spans as many rows as it is tall (measured with `ResizeObserver` as its embed loads), so posts pack like masonry while keeping date order. Do not switch to CSS multi-column layout. Embeds load as they approach the viewport. Under each embed show only its subtle publication age; the post itself says which models it covers. Source `kind`, `comparisonModelIds`, caveats, scope and editorial rationales stay as provenance in the JSON linked from this pipeline; the page does not render them. The executable provider catalog remains separate. Future runtime consumption should bundle a pinned, reviewed exact-model recommendation policy, not fetch it at startup. Do not change model IDs, capabilities, provider availability, user ordering, selected models, active sessions, or hide low-tier or not-shown models as a side effect of an editorial refresh. The page covering models outside Happy's catalog does not add them to the picker. Coordinate any runtime policy update separately and preserve local overrides. ## Validation and handoff Run `pnpm test` and `pnpm build` (in the sandbox, set `TMPDIR` to a directory under `.context/`). Tests must enforce unique IDs, valid X URLs, reciprocal model/source references, no more than four posts per model, exact comparison membership, supported notes, automatic widget loading, and useful blocked-widget fallback. They must also enforce the chart shape: only S, A, B and F; one to three S models, each with a lane and at least one in Frontier; no lane outside S; at most seven shown models grouped in tier order; no empty tier row rendered; the `<lab>/<model-slug>` ID format; every major lab present on the chart or in `notShown`; each `notShown` entry with a reason and note, a superseded model's successor from the same lab, and an outranked model's `by` on the chart; and no source citing a model that is not shown. And they enforce the `leadCheck` rules: only on the leader, a valid non-future date, `heldAgainst` models on the page, and sources that cover every challenger. And the card rules: every point linked to one of the model's own posts and short enough for the card, every headline from the same board with its rank, effort and score matching the board post and #1 and #2 consistent, `releasedAt` matching the release post's snowflake, the badge's 14-day cutoff, the S row's label and card cells, and no popover or filter. And the context rules: every card point or lead note with a number names a metric, no publisher prefix ("Artificial Analysis", "Theo Browne", "Cognition") or "AA", no fraction without what it counts, and no point that is just a number with a label. And the S row's nested table: tier letter plus one cell per row, one header per lane, one body row, and the card-gap arithmetic in the CSS. And the wall and price rules: at most eight wall posts with unique status IDs, none quoting or replying to another post or quoted by one, newest first on the page, the lead check and the Chores pick's value first in `wall`, every shown model on the wall; no Agents on Rails source or point; no price chip, price label or `price`/`pricing` data. Check `/model-benchmarks` and `/model-benchmarks/`, static canonical/social metadata, and existing redirects. Visually inspect screenshots in `.context/blueprints/`: the full page with real loaded embeds, the S and A rows up close (with the measured card x positions), a card with the new-model badge, a claim link in its hover state, all cards, the posts wall (check two columns, date order, no post split or embedded twice), tablet, blocked-widget fallback, and mobile. Cross-origin embeds paint only inside the viewport, so for a full-page capture grow the viewport to the page height rather than using a full-page screenshot. Remove stale captures when the design changes. The current site is light-only; do not invent a dark theme. Clearly distinguish real embed screenshots from mocked failure/success test fixtures. Hand off the tier table (S as Frontier and Chores), every model added, moved or removed with its reason (including supersession), each lab's latest release and where it landed, every selected post with author/date/support/category, rejected or replaced candidates and reasons, exact verification results, screenshots, and remaining uncertainty. ## Publishing Kirill has given standing authorization to publish benchmark refreshes (requested every two days). After `pnpm test` and `pnpm build` pass and you have checked the screenshots, commit and push to `main`, which deploys the public site. Rebase onto a freshly fetched `origin/main` first. The authorization covers only this page: `src/model-benchmarks.json`, `src/modelBenchmarks.ts`, `src/ModelBenchmarksPage.tsx`, `src/model-benchmarks.css`, their tests, and this skill. Any other change needs explicit approval. A no-material-change run commits nothing.