self-consistency

# self-consistency Don't trust a single greedy chain of thought on a hard problem. Generate several INDEPENDENT solutions and take the one they agree on. Agreement across diverse reasoning paths is a strong correctness signal. ## Why this exists (evidence) - Self-consistency (sample diverse reasoning paths, marginalize to the most consistent answer) beats single greedy chain-of-thought by large margins: +17.9pp on GSM8K, +11.0 SVAMP, +12.2 AQuA, +6.4 StrategyQA over the same model. Source: Wang et al., arXiv:2203.11171. - The gain is free of retraining: it is pure test-time compute spent where being wrong is costly. ## When to use - A bug whose root cause is unclear (sample several diagnoses). - A design/architecture call with multiple defensible options. - Math, logic, estimates, migrations, or anything where a confident-but-wrong answer is expensive. - NOT for cheap/obvious steps: it multiplies cost. Use it where the stakes justify the spend. ## The method 1. Restate the problem and the answer format (so samples are comparable). 2. Generate N INDEPENDENT attempts (3-5 typical), each reasoning from scratch and from a different angle. Do not let them see each other. 3. Extract the final answer from each. 4. Take the answer that recurs most (majority / consistency vote). Tie or wide spread = low confidence -> get more samples or escalate to the user. 5. Report the vote: the chosen answer, the count (e.g. 4/5), and the dissent (what the minority said and why it might be right). ## How to run it - Inline: write N short independent analyses yourself, then vote. - Scaled: use a Workflow to fan out N agents on the same prompt with `parallel(...)`, collect their structured answers, and majority-vote in code. Route the sampling to a cheaper model (see model-router); keep the final adjudication sharp. - For code: sample N candidate patches, then prefer the one that passes tests AND recurs across samples. ## Composes with - `model-router`: sample cheap, adjudicate sharp. - `adversarial-verify` / `testsmith`: verify the winning answer, don't just trust the vote. - `run-cost`: N samples = N x cost; budget before fanning out. ## Honest limits - It multiplies token cost by N. Only worth it when an error is more expensive than the extra samples. - Consensus is a heuristic, not proof: all samples can share the same wrong bias. Pair with verification for anything critical. - The cited gains are on reasoning benchmarks; code-task transfer is plausible but measure your own.