big-context-analysis

name	big-context-analysis
description	The map-reduce + blackboard pipeline that lets a small-context model digest inputs far beyond one attention window — a codebase, a long document, a corpus, a multi-part idea — and state only evidence-anchored, citation-checked conclusions. Use when / Trigger: executing or resuming an analyst run (/analyst:digest, /analyst:resume, /analyst:show), or any analysis input exceeds one 200-line chunk.

Big-context analysis

A frontier model holds a whole corpus in one window; the executor here cannot. This skill replaces the one-window act with staged structure: every model invocation sees one attention-budget unit, every finding is a typed row on a durable blackboard, and no conclusion may be stated without citing finding ids a script verifies. Rationale: docs/plugins/analyst.md (marketplace repo).

The six phases

#	phase	invocation shape	writes	phase file
1	chunk	script only — no model call	chunks/ + manifest.json (id, source span, sha256, overlap)	references/phase-1-chunk.md
2	map	ONE chunk per mapper call, fixed extraction JSON	maps/C-*.json	references/phase-2-map.md
3	merge	mechanical SQL — dedup by (type, key); no model call	findings rows + summaries/0-*.json	references/phase-3-merge.md
4	reduce	one merger call over exactly TWO summaries, log2 depth	summaries/<L>-*.json	references/phase-4-reduce.md
5	conclude	one concluder call per root theme + N citation votes	conclusions rows + verdict.json	references/phase-5-conclude.md
6	strategize	only when the question matches the §Create action regex	strategy.md	references/phase-6-strategize.md

The finding schema (typed, dedup-keyed — the blackboard row)

column	rule
id	F-0001… — the citation target for every conclusion
run_id, chunk_id	provenance back to a manifest digest
type	closed enum: claim / entity / risk / dependency / open-question / contradiction
key	normalized dedup key — entity name, file path, claim subject, X -> Y
body	one paragraph max, no newlines or tabs
anchor	source span path:start-end — blank is rejected at insert (schema CHECK, not judged)

Mappers never emit contradiction; that type is minted mechanically at merge when one (type, key) carries differing bodies, and from the merger's cross-input relations slot.

The conclusion slot card

slot	rule
claim	one sentence
finding_ids	≥1, script-verified to exist (§CiteCheck) — a dangling id fails before any model judges
confidence	high / medium / low — set by vote agreement, never self-reported
dissent	non-empty whenever votes split or a contradiction shares a cited key
load_bearing	SQL rule: cites a risk/contradiction finding OR spans ≥3 chunks → vote floor 3

Sizing (M1 — computed from countables, never self-sized)

input size	chunk size	map fan-out	tier	conclusion votes
≤ 9 chunks	200 lines, 30-line overlap	1 mapper call per chunk, parallel batches of 8	T1	1
10–99 chunks	same	same	T2	3
100–2000 chunks	same	same	T3	5
> 2000 chunks	—	—	halt: stop and report, ask for narrower paths	—

Contradiction-dense board (contradictions × 20 > findings) bumps the tier to T3 at merge. Vote splits: T2 2–1 → +2 votes once; T3 ≥2 of 5 → one sonnet-class pass supersedes (M7 R2).

Laws (always in force)

1. One chunk per mapper invocation; one summary pair per merger invocation; one theme per concluder invocation. No invocation ever loads the corpus whole.
2. Cite-or-die: a conclusion or strategy action citing an id that fails §CiteCheck is laundering — the run halts; the text is never reworded past the check.
3. Conservation: every finding id in a merger's two inputs appears in exactly one output theme; ids are never invented. jq-checked after every merge.
4. Checkpoint after every phase (§Checkpoint): artifacts to disk first, store rows second.
5. A failed phase blocks the run by construction (no v_next_step row) — report it; never improvise past it.
6. A chunk failing map validation twice is logged in maps/*.failed, never silently skipped; failed chunks × 10 > total chunks → halt.

Failure-mode absorbers

failure	absorber
chunk-boundary blindness	fixed 30-line overlap windows + the merger's cross-input relations slot over adjacent summaries
blackboard bloat	typed dedup keys + per-type cap (500 rows) → M7 stop-and-report, never silent truncation
conclusion laundering	§CiteCheck scripts — dangling ids fail mechanically before any model judges content
map essays (schema drift)	jq schema-validate + one retry per chunk; the second failure is logged, counted, capped

Files in this skill

file	load when
references/run-state.md	creating, resuming, checkpointing, failing, or reporting a run
references/phase-<n>-<slug>.md	executing that phase — one at a time, never bulk