twda

name	twda
description	Analyze the TWDA (Tournament Winning Deck Archive) to refresh the site's slow-moving content — best-cards stats and top-archetype identification via deck clustering. Use when recomputing best cards, tier assignments, meta statistics, or classifying TWD decks into archetypes.

TWDA analysis

Turns the TWDA into the numbers the site's content needs (see TODO.md P2).

Data

• The TWDA (~4,500 decks since 1994), via the krcg library models: https://static.krcg.org/data/v5/twda.json decoded into krcg.models.Deck (daily-cached in ~/.cache/codex-twda/, krcg's bundled snapshot as offline fallback). Deck entries (CardInDeck) carry kind, types and disambiguated names — unique_name keeps group variants distinct, twda.merge_key() collapses them (re-grouped star vampires define the same archetype).
• Anchors: templates/archetypes/*/*.json each carry the TWDA id of the page's example deck — free ground-truth labels (~90, of which ~27 live, i.e. non-archive). twda.load_anchors() maps TWDA id → archetype slug + section.

scripts/twda.py is the shared data layer. The krcg library (same author as this site) is the intended toolkit for TWDA work — prefer extending on top of it; krcg.analyzer also offers set-overlap primitives (played/affinity/ build_deck) useful for deck-building suggestions.

Scripts

All in scripts/, self-contained (PEP 723, uv run) — the site gains no dependency.

uv run scripts/best_cards.py --since 2020-01-01 --top 20   # play stats by card type
uv run scripts/cluster.py --since 2021-07-01 --out c.json  # archetype clustering
uv run scripts/cluster.py --anchors-nn                     # vector-space sanity check
uv run scripts/review_page.py c.json -o review.html        # owner review/editor page

review_page.py renders the owner's review & classification editor (publish as an artifact): group naming, bulk deck moves, group create/delete, variant-of links (a variant group nests under its main, TOC reordered, qualifying counts combined on the main), representative stars, criteria greying (archetypes index rule: 20+ players twice in the last 3 years — proven groups show their count in the TOC), NC/CC badges on qualifying national/continental championship wins (regex on event names; qualifiers/side events excluded), proposed variant splits, and an Export button whose JSON is the input for archetype-page generation. Edits persist in localStorage — regenerating the page with identical clustering (same data + params ⇒ identical C-refs; verify when in doubt) keeps them valid.

Writing archetype pages

Once the classification/tiering is settled and the owner has picked representatives, the site's archetype pages get written or refreshed. The authoritative brief — the template contract AND the prose-style rules the owner enforces (no deckbuilding copy counts in the generic description; never explain what a card does, since the site hovers the card text and readers know the cards; get the strategy right, not just the card text) — is references/archetype-page-style.md. Read it before writing any page, and pass it to any subagent you delegate a page to. Decklist JSONs come from scripts/page_json.py <twda_id>.

TWDA refresh workflow (the working tool)

1. uv run scripts/cluster.py --since <5 years back> --out data/clusters.json
2. uv run scripts/review_page.py data/clusters.json -o review.html, publish as an artifact (keep the same file path to keep the URL). The page embeds all decklists (hover) with live rare-card / odd-count / missing-staple signals against the deck's current group.
3. The owner reviews and refines in the page (names, moves, splits, variants, representatives), then Export. Save the export's editor_state as data/review-<date>.editor-state.json and rebuild the canonical file: uv run scripts/apply_review.py data/clusters.json <state> --generated <date> -o data/classification.json. Commit both. classification.json is the ground truth: the owner's labels are the most valuable asset this skill produces (first pass 2026-07-05: 89 archetypes + 30 variants, 1,083 decks, 51 proven). The editor_state can also be re-imported into a page generated from the SAME clustering run (refs must match). scripts/benchmark.py scores any run against the labels (baseline: ARI 0.861 groups / 0.687 variant-merged) and prints the tier table (--tiers).
4. Next refresh: uv run scripts/classify.py --refresh [--out new.json] — don't re-cluster. It embeds labeled + new decks in the same space, assigns each new deck to the nearest group centroid (threshold calibrated from leave-one-out validation: 94.3% exact group / 98.6% archetype accuracy on the 2026-07 labels), and HDBSCANs the leftovers into proposed novel archetypes. The owner confirms the deltas; fold them into the editor state and re-run apply_review.py. Each confirmed pass grows the labels.
5. Any pipeline tuning must be scored against data/classification.json (ARI / anchor checks) before being adopted — the labels are the benchmark, never a thing to silently overwrite.

Clustering pipeline (the hard part — still experimental)

Card-count vectors → sublinear TF + IDF (mutes staples like Villein/Deflection) → crypt and library blocks L2-normalized separately then weighted (--crypt-weight, default 0.5 — crypt defines archetypes, don't let the 80-card library drown it) → TruncatedSVD (--components) → HDBSCAN (default) or agglomerative/cosine (--method agglo --threshold). Vampire group variants are merged by default (--keep-groups to disable).

Evaluation, printed by every run: cluster count and noise share, anchors landing in noise, and — the key failure signal — clusters merging several live anchors (distinct archetypes should not share a cluster). --out file.json dumps assignments for downstream use.

If auto-clustering stalls, fall back to human-in-the-middle: propose nearest anchors + characteristic cards per new deck, let the owner confirm (see TODO.md).

Status 2026-07-05: the defaults are the winning config of the first sweep — HDBSCAN(min_cluster_size=5, min_samples=2) on 2020+ (1,430 decks) gives 107 clusters, 17% noise, 41/44 anchors clustered, zero live-anchor merges, and coherent archetypes down to size-5 clusters. Agglomerative assigns everything but merges close archetypes — candidate for attaching noise decks later, not the primary method. Owner review of the 5-year run confirmed quality overall.

Within-cluster variants (e.g. an ally-rush module inside the Lasombra Oblivion bleed cluster) are NOT separable by global re-weighting — a card-type-totals block (--type-features) changed nothing at 0.1 and only added noise at 0.15. What works is local sub-clustering (--subcluster, default 0.8): rebuild the matrix over one cluster's decks so the IDF is local, staples flatten, and the variant modules emerge — it reproduced the owner's manual split of C5 exactly and correctly left cohesive clusters alone. Details and next steps in TODO.md P2.