| name | auditing-intelligence-context-fit |
| description | Audits the fit between a model's reasoning capability and the complexity of the context it receives. Use when an AI system is underperforming despite good retrieval, when teams are unsure whether to upgrade their model or improve their context pipeline, or when diagnosing the "long context and weak reasoning equals harmful output" pattern in production agent systems.
|
Auditing Intelligence-Context Fit
The Multiplicative Model
output_quality = reasoning_tier × context_quality
This is multiplicative. A mismatch in either dimension degrades output non-linearly:
| Reasoning Tier | Context Quality | Output |
|---|
| High | High | Excellent |
| High | Low | Mediocre (model can't compensate for bad retrieval) |
| Low | High | Mediocre (model can't leverage complex context) |
| Low | Low | Poor |
| Low | Very High (long, complex) | Harmful — model hallucinates connections in complex context it can't parse |
The last row is the "long context + weak reasoning = harmful output" failure. It's worse than simply poor because the model generates confident-sounding wrong answers.
The Degradation Curve
As context complexity increases beyond a model's reasoning ceiling:
- Phase 1 — Normal operation: Model handles context well; output quality matches expectation
- Phase 2 — Reasoning plateau: Model is at capacity; quality stops improving even as context improves
- Phase 3 — Context poisoning: Additional context overwhelms the model; it begins drawing spurious connections; quality degrades
- Phase 4 — Expensive RAG fallback: System compensates by adding more retrieval passes, reranking, and chain-of-thought prompting — all of which add cost without fixing the root mismatch
Symptom of Phase 3–4: "We added better context and the answers got worse, so we added more prompting."
flowchart LR
P1["Phase 1\nNormal Operation\nQuality ✓"] -->|complexity grows| P2["Phase 2\nReasoning Plateau\nQuality flat"]
P2 -->|more context added| P3["Phase 3\nContext Poisoning\nQuality ↓\nSpurious connections"]
P3 -->|team compensates| P4["Phase 4\nExpensive RAG Fallback\nCost ↑↑ Quality unchanged"]
style P1 fill:#dcfce7,stroke:#22c55e
style P2 fill:#fef9c3,stroke:#eab308
style P3 fill:#fed7aa,stroke:#f97316
style P4 fill:#fee2e2,stroke:#ef4444
The Fit Matrix
quadrantChart
title Intelligence-Context Fit — Output Outcomes
x-axis "Low Model Tier (Haiku)" --> "High Model Tier (Opus)"
y-axis "Simple Context (L1–2)" --> "Complex Context (L4–5)"
quadrant-1 "Optimal: excellent output"
quadrant-2 "Over-resourced: high cost, correct output"
quadrant-3 "Cost-efficient: correct output"
quadrant-4 "Harmful: confident wrong answers"
Haiku + Factual: [0.15, 0.15]
Sonnet + Relational: [0.5, 0.5]
Opus + Causal: [0.85, 0.85]
Haiku + Causal: [0.15, 0.85]
Opus + Factual: [0.85, 0.15]
The Audit Framework
Step 1: Classify Context Complexity
Rate the complexity of the context being passed to the model on this scale:
| Level | Description | Example |
|---|
| 1 — Factual | Single documents, direct lookup | "What does policy X say?" |
| 2 — Comparative | Multiple documents, explicit comparison | "Compare options A, B, C" |
| 3 — Relational | Cross-document entity relationships | "What teams are affected by X?" |
| 4 — Temporal | Events across time, sequence reasoning | "How did this evolve over 6 months?" |
| 5 — Causal | Multi-hop causal chains | "What chain of decisions led to Y?" |
Step 2: Assess Model Reasoning Tier
| Tier | Model Examples | Appropriate Context Complexity |
|---|
| Basic | Claude Haiku 4.5 | Level 1–2 |
| Mid | Claude Sonnet 4.6 | Level 1–4 |
| Advanced | Claude Opus 4.6 | Level 1–5 |
Note: Model capability evolves rapidly. Retest your tier assignments when new model versions release.
Step 3: Identify the Mismatch
Scenario A: Context too complex for model tier
- Symptom: Confident but incorrect answers; spurious causal claims; inconsistent reasoning across runs
- Fix: Upgrade model tier OR simplify context (decompose into sub-queries at lower complexity levels)
Scenario B: Model over-qualified for context
- Symptom: Correct output but unnecessarily high cost and latency
- Fix: Downgrade model tier for this query class; reserve advanced models for Level 4–5 queries
Scenario C: Context quality is the bottleneck, not model
- Symptom: Upgrading the model doesn't improve output; answers are plausible but missing key facts
- Fix: Improve retrieval pipeline (check
diagnosing-rag-failure-modes); model is not the constraint
Scenario D: Context volume exceeds model window effectively
- Symptom: Model ignores content from the middle of long contexts; only uses beginning/end
- Fix: Apply windowed compression (see
temporal-reasoning-sleuth); reduce context length, not model
Step 4: Prescribe the Fix
| Mismatch | Fix |
|---|
| High complexity, low-tier model | Upgrade model OR decompose query |
| Low complexity, high-tier model | Route to lower-tier model |
| Poor retrieval → poor context | Fix retrieval pipeline (not the model) |
| Context too long for model | Apply windowed compression |
| All tiers failing on one query type | Structural mismatch — needs hybrid architecture |
Common Anti-Patterns
Anti-Pattern: Haiku + Organizational History
Using a basic-tier model on Level 4–5 context complexity. The model confidently fills gaps it cannot reason over. The output looks plausible but contains fabricated causal connections.
Fix: Reserve causal/temporal queries for Sonnet or Opus tier. Use Haiku only for factual lookup (Level 1–2).
Anti-Pattern: Opus on Simple RAG Lookups
Using an advanced-tier model for point-query factual retrieval. Correct output, but 5–10× the cost and latency of an appropriate tier.
Fix: Route Level 1–2 queries to Haiku; reserve Opus for Level 4–5 queries where reasoning depth matters.
Anti-Pattern: Compensating with More Prompting
When output quality degrades, adding more system prompt, chain-of-thought instructions, and few-shot examples to compensate for a model-context mismatch. This adds cost and complexity without fixing the root cause.
Fix: Identify which step of the audit framework is the actual mismatch and fix it there.
Output: Fit-Audit Report
INTELLIGENCE-CONTEXT FIT AUDIT
================================
System / Query Class: [what you're auditing]
Context Complexity Level: [1–5] — [description of why]
Current Model Tier: [Haiku / Sonnet / Opus]
Observed Symptom: [what's going wrong]
Mismatch Type: [Scenario A / B / C / D from Step 3]
Root Cause:
[One paragraph explaining the mismatch]
Recommended Fix:
[Specific action — upgrade model / decompose queries / fix retrieval / compress context]
Estimated Impact:
[Quality improvement expected] / [Cost change expected]
Related Skills:
[Link to diagnosing-rag-failure-modes / designing-hybrid-context-layers as applicable]
