limits-of-expertise-and-prediction

name	limits-of-expertise-and-prediction
description	Audits an analytic judgment's use of theory, generalizations, and prediction limits — data- vs. concept-driven method and the mosaic fallacy — before the judgment is finalized.
kind	skill
status	ready
provenance	{"principles":["P018","P019","P041","P048","P049","P052","P076"],"claims":["C00162","C00163","C00218","C00219","C00220","C00221","C00277","C00278","C00279","C00283","C00284","C00285"],"evidence":[],"source_anchors":[]}

Limits Of Expertise And Prediction

Purpose

Review whether an analytic judgment stays inside what its own method can honestly support: classified consistently as data-driven or conceptually-driven and argued with the confidence proper to that kind (P048), any theory, generalization, or covering law applied to the case checked against its antecedent conditions and stated boundary conditions rather than assumed to hold (P076, P041, P052), imaginative or theory-driven reasoning kept in a managed trade-off rather than left unpruned (P019) or fought out fact-by-fact instead of framework-by-framework (P018), and a call for more collection recognized as no substitute for the hypothesis-first work of analysis (P049). This skill audits the judgment's respect for the limits of theory, expertise, and prediction — not the substantive conclusion itself.

When to use

• An interpretive dispute over an actor's intentions or behavior is being argued by citing one or two facts or the most recent behavior, rather than debating the rival general images or frameworks each side brings to the same evidence (P018).
• A hypothesis or forecast emerged from open-ended, imaginative, or speculative reasoning, and it is unclear whether implausible branches were pruned before they multiplied confusion (P019).
• A general political theory, doctrine, or "lesson" is applied to a specific case with no elaboration into a time-bound, checkable set of early-warning indicators (P041).
• It is unclear whether the judgment is a data-driven determination resting on an explicit, teachable model or a conceptually-driven one resting on an implicit mental model — and whether its stated confidence matches which kind it actually is (P048).
• The judgment or its authors argue that more collection, rather than better analysis, is what would clarify an unclear picture (P049).
• A generalization, base rate, or outside-view estimate is applied to the case with no boundary conditions or reference class stated (P052).
• A theory or covering law is invoked to explain or predict the case, and whether its antecedent conditions actually hold in this case has not been checked (P076).

Procedure

1. Establish what kind of analysis the judgment claims to be — a data-driven determination or a conceptually-driven judgment resting on an implicit or explicit theory, model, or framework — and identify which theory, generalization, or covering law is doing the inferential work (P048).
2. Check that the judgment's expressed confidence matches which kind of analysis it actually is: flag a conceptually-driven judgment, where the outcome depends on an implicit, contested mental model, that claims the objective certainty proper only to a data-driven determination built on a correct, teachable model (P048).
3. Where a theory, generalization, or covering law is applied to explain or predict the case, verify its antecedent conditions are shown to hold against the case's actual facts, rather than assumed to hold because the theory is well established elsewhere (P076).
4. Where a general theory or historical "lesson" is invoked, check that it has been elaborated into a time-bound, checkable set of early-warning indicators — an unelaborated generalization economizes thought but specifies no time frame for when its prediction should occur (P041).
5. Where a boundary generalization, base rate, or outside-view estimate is applied to the case, confirm the boundary conditions under which it holds and the reference class chosen — and why — are stated, rather than the estimate being asserted unconditionally (P052).
6. If an interpretive dispute is being argued by citing one or two facts or the most recent behavior, check that the rival general images or frameworks behind each side's interpretation are surfaced and debated instead — a dispute that is actually over frameworks will not be resolved by facts alone (P018).
7. If the judgment or its hypothesis set arose from open-ended or imaginative reasoning, check that plausibility pruning was applied — speculative branches cut once they exceeded the bounds of probability — rather than left to multiply confusion (P019).
8. If the judgment or its authors call for more collection as the fix for an unclear picture, check whether the real gap is analytic — a hypothesis is needed to organize the pieces already in hand — rather than a data shortfall that the mosaic theory promises collection alone will resolve (P049).
9. Emit findings highest-impact first, each naming the reasoning flaw, the failure it enables, the applicable principle, the corrective, and the residual uncertainty — never issuing the substantive judgment itself.

Inputs

• The analytic judgment or forecast under review, including which theory, generalization, doctrine, or covering law, if any, it invokes.
• Whether the judgment presents itself as data-driven or conceptually-driven, and what mental model or framework is doing the inferential work.
• Any boundary conditions, reference class, or antecedent conditions the judgment states, or omits, for a generalization or theory applied to the case.
• Any argument, explicit or implicit, that more collection would resolve the analytic problem.
• The specific facts or "most recent behavior" cited in an interpretive dispute, and the rival interpretation being argued against.
• Whether the hypothesis set was generated through open-ended or imaginative reasoning, and whether any plausibility check was applied to it.

Output

Per finding: name the reasoning flaw (a misclassified data- versus concept-driven judgment, an unchecked covering-law antecedent, an unelaborated generalization, a missing boundary condition or reference class, a fact-only argument in a framework dispute, unpruned speculation, or the mosaic fallacy) and the failure it enables, apply the principle by ID and the correction (reclassify the analysis and its warranted confidence, check or state the antecedent conditions, elaborate early-warning indicators, state the boundary conditions and reference class, debate the rival frameworks, prune the hypothesis space, or redirect from collection to analysis), state the residual uncertainty the correction leaves, and end with a concrete next step. Order findings highest-impact first. Never issue the substantive judgment in place of this structure.

Anti-patterns to flag

• A dispute over an actor's intentions or behavior fought fact-by-fact or by citing the most recent behavior alone, with the rival general images or frameworks behind each side's interpretation never named or debated (P018).
• Imaginative or speculative hypotheses left unpruned — carried forward with no plausibility check — in the name of open-mindedness, letting confusion substitute for judgment (P019).
• A general theory, doctrine, or historical "lesson" applied to the case as if self-sufficient, with no elaboration into time-bound, checkable early-warning indicators (P041).
• A conceptually-driven judgment — resting on an implicit, contested mental model — argued with the confidence proper only to a data-driven determination built on a correct, teachable model (P048).
• A call for more collection offered as the fix for an unclear picture, when the real gap is that no hypothesis yet exists to organize the pieces already in hand (P049).
• A generalization, base rate, or outside-view estimate applied to the case with no boundary conditions stated and no reference class named, as though the subjective-objective tension did not exist (P052).
• A theory or covering law invoked to explain or predict the case with its antecedent conditions assumed rather than checked against the case's actual facts (P076).

References

See ../../references/analytic-method-principles-index.md for the full principle catalogue. For adjacent concerns, see the sibling skills: probabilistic-judgment-and-calibration owns the coherence and calibration mechanics once a reference class or outside-view estimate is chosen; competing-hypotheses-and-diagnostic-evidence owns building and testing the hypothesis set once picture-first analysis has produced one; structured-analytic-techniques owns the mechanics of brainstorming and other techniques that generate the ideas this skill checks for plausibility pruning.

Provenance

Derived solely from P018, P019, P041, P048, P049, P052, and P076 (Psychology of Intelligence Analysis; Expert Political Judgment: How Good Is It? How Can We Know? — both distillation-only; see the frontmatter above for the full claim, evidence, and source-anchor list).