analytic-collaboration-training-and-process

name	analytic-collaboration-training-and-process
description	Audits the training, team, and review process behind an analytic judgment — devil's advocacy, pre-publication review, postmortems — invoke to review that process, not the judgment itself.
kind	skill
status	ready
provenance	{"principles":["P001","P008","P010","P015","P022","P024","P040","P058","P063","P071","P081"],"claims":["C00002","C00008","C00009","C00016","C00017","C00018","C00019","C00020","C00021","C00022","C00051","C00052"],"evidence":[],"source_anchors":[]}

Analytic Collaboration Training And Process

Purpose

Audit the training, team design, and institutional review process behind an analytic judgment or forecast — not the judgment's content, but whether the analyst and the shop around them were set up to reason well. This skill checks whether the reasoning process itself was trained and coached rather than only the writing, whether competing-view procedures were institutionalized rather than staged for show, whether management mandated a systematic method on the issues that warranted it, and whether postmortems and measurement stayed rigorous and unbiased. It audits the scaffolding around the judgment, never the judgment itself.

When to use

• A judgment or forecast came out of a team, and it is unclear whether that team was actually psychologically safe, diverse in perspective, and organized around a shared purpose — or produced groupthink dressed up as consensus.
• A consensus formed after a designated devil's-advocate or similar role weighed in, and it is unclear whether the agreement is genuine convergence (people who previously held a different view coming around against their own position or interest) or an artifact of the role itself manufacturing false confidence that dissent was heard.
• A high-stakes or deception-prone issue was carried on a single-outcome analysis, and whether management should have mandated — or did mandate — a systematic process such as Analysis of Competing Hypotheses needs checking.
• A draft is moving through pre-publication review, and the review needs to confirm it actually interrogated the analyst's unstated assumptions, the alternatives considered and rejected, and what would change the analyst's mind — ideally by critical thinkers from outside the author's branch.
• An intelligence failure, a stale estimate, or a suspiciously well-confirmed conclusion needs diagnosing for whether negative evidence went unweighted or an organizational incentive or culture problem is at play, rather than defaulting to blaming one analyst's psychology.
• An analyst-development or training plan is under review, to check it targets the analyst's own reasoning process — coaching, postmortems on both failures and successes, retrospective evaluation, practice by doing — rather than only writing craft or one-off classroom instruction.
• The analytic process itself is being designed or audited: whether it institutionalizes competing-view procedures, periodic ground-up re-examination, honest expectation-setting with consumers about analysis's limits, and blind independent coding wherever results are measured.

Procedure

1. Identify which process object is under review — a team's collaborative setup, a group's convergence on a judgment, a management decision about formalizing method, a training/development plan, an organizational failure being diagnosed, or a measurement/coding exercise — before applying the checks below.
2. Check the team or group behind the judgment for psychological safety, a shared "we" purpose, and genuine diversity of perspective rather than ability alone, since these — not raw individual skill — predict the aggregation gains and open-mindedness a good process should show (P015).
3. If a devil's-advocate or similar role was used, do not credit the resulting consensus at face value: check whether people who previously held a different view converged independently, especially against their own prior position or self-interest, and discount any convergence the role itself could have manufactured by letting the group feel falsely open-minded (P008).
4. For a high-stakes or deception-prone issue, check whether management rejected a single-outcome analysis and mandated a systematic process such as Analysis of Competing Hypotheses — requiring the alternatives considered, why each was judged less likely, and an explicit statement that events may not turn out as expected (P010).
5. Trace whether the analyst's or team's training and self-examination were directed inward at their own thinking and reasoning process — not only at organizational procedure, method, or the substantive topic — and whether that training was coached and learned by doing rather than delivered as classroom instruction alone (P001, P022, P071).
6. When diagnosing an intelligence failure, a stale confirmed estimate, or a lapse in evidence-weighing, check whether the organization's incentive structure or culture — not only one analyst's psychology — let negative evidence (the dogs that did not bark) go unweighted, and whether training and discipline were in place to counter that (P024).
7. Check that management supports periodically re-examining key problems from the ground up rather than relying on incremental adjustment alone, and that consumers were educated about analysis's limitations as well as its capabilities rather than sold false certainty (P040).
8. Confirm the analytic process itself institutionalizes procedures that surface competing views — analytic debate, devil's advocacy, competitive analysis, peer review, outside expertise — and rewards weighing multiple hypotheses over accepting the first credible one (P058).
9. For a draft under pre-publication review, confirm the review explicitly questioned the analyst's mental model — what unstated assumptions underlie the judgment, what alternatives were considered and why rejected, what would change the analyst's mind — and that reviewers included capable critical thinkers from outside the author's branch, not only same-branch colleagues who share the mind-set (P063).
10. If a measurement or scoring exercise is in scope, confirm coding was performed by independent coders blind to the hypotheses, so the analyst's own biases could not leak into the result (P081).

Inputs

• The analytic judgment, forecast, or draft under review, and whether it was produced by an individual or a group.
• What is known about the team's setup: size, roles (including any devil's-advocate or similar role), psychological safety, and diversity of perspective.
• Any record of who held which view before and after a consensus formed, including whether convergence ran against a person's own prior position or self-interest.
• The issue's stakes and deception risk, and whether a systematic method such as Analysis of Competing Hypotheses was mandated, considered, or skipped.
• The analyst's or team's training and development history — what it targeted and how it was delivered — and any record of postmortems or retrospective evaluation.
• The organization's process design: which competing-view procedures are institutionalized, how consumers are briefed on analysis's limits, and whether periodic ground-up re-examination happens.
• The pre-publication review record — who reviewed the draft and what was asked — and, if scoring or coding is in scope, whether coders were independent and blind to the hypotheses.

Output

Per finding: name the process flaw (a devil's-advocate role credited at face value instead of weighed for genuine independent convergence, an unmandated single-outcome analysis on a high-stakes or deception-prone issue, training aimed at writing instead of the reasoning process, a pre-publication review that never questioned the analyst's mental model, non-blind or non-independent coding, or a competing-view procedure never actually institutionalized), apply the correction implied by the relevant principle (weigh independent convergence rather than the role's face value, mandate a systematic method, redirect training and coaching inward and make it practice-based, bring in outside critical thinkers, use blind independent coders, institutionalize the missing procedure), state the residual uncertainty the correction leaves, and end with a concrete next step. Order findings highest-impact first. Never substitute a bare pass/fail verdict for this structure, and never produce the substantive analytic judgment yourself.

Anti-patterns to flag

• A devil's-advocate or similar role credited as proof the group considered dissent, with the false confidence of having been open-minded substituted for evidence of genuine independent convergence (P008).
• A team or group lacking psychological safety or genuine diversity of perspective, credited with an aggregation gain its composition could not structurally have produced (P015).
• A high-stakes or deception-prone issue carried on a single-outcome analysis with no mandate for a systematic method like Analysis of Competing Hypotheses, no stated alternatives, and no acknowledgment that events may diverge from expectation (P010).
• Training and self-examination aimed only at writing quality, organizational procedure, or the substantive topic, never turned inward on the analyst's own thinking and reasoning (P001, P022).
• Analytic-thinking development delivered as one-off classroom instruction with no coaching and no practice-by-doing (P071).
• An intelligence failure or a suspiciously well-confirmed conclusion blamed solely on one analyst's psychology, with the organization's incentive structure, culture, or neglected negative evidence never examined (P024).
• Analysis run only incrementally, with key problems never re-examined from the ground up, and consumers never told what analysis cannot do (P040).
• A process design with no institutionalized competing-view procedure — no analytic debate, devil's advocacy, competitive analysis, peer review, or outside expertise sought — and the first credible hypothesis rewarded over weighing several (P058).
• Pre-publication review performed only by same-branch colleagues who share the analyst's mind-set, without asking what assumptions were unstated, what alternatives were rejected and why, or what would change the analyst's mind (P063).
• Coding or scoring performed by coders who know the hypotheses, or by the analyst themselves, letting bias leak into the measurement (P081).

References

See ../../references/analytic-method-principles-index.md for the full principle catalogue. For adjacent concerns, see the sibling skills: competing-hypotheses-and-diagnostic-evidence runs the Analysis of Competing Hypotheses matrix itself — diagnosticity weighing and refutation-seeking — once this skill's review confirms formalization was warranted; structured-analytic-techniques selects and sequences which technique to apply; assumptions-framing-and-analytic-writing reviews an individual draft's explicit linchpin assumptions and its separation of fact from opinion; cognitive-biases-and-dual-process-reasoning covers the individual analyst's bias catalogue that the team, training, and review design in this skill exist to counter.

Provenance

Derived solely from P001, P008, P010, P015, P022, P024, P040, P058, P063, P071, and P081 (this package's six analytic-tradecraft sources: Psychology of Intelligence Analysis; Thinking, Fast and Slow; Superforecasting; Expert Political Judgment: How Good Is It? How Can We Know?; Perception and Misperception in International Politics; and the CIA Sherman Kent School's Tradecraft Primer — all distillation-only; see the frontmatter above for the full claim list).