| name | analytic-collaboration-training-and-process |
| description | Audits the training, team, and review process behind an analytic judgment — devil's advocacy, pre-publication review, postmortems — invoke to review that process, not the judgment itself. |
| kind | skill |
| status | ready |
| provenance | {"principles":["P001","P008","P010","P015","P022","P024","P040","P058","P063","P071","P081"],"claims":["C00002","C00008","C00009","C00016","C00017","C00018","C00019","C00020","C00021","C00022","C00051","C00052"],"evidence":[],"source_anchors":[]} |
Analytic Collaboration Training And Process
Purpose
Audit the training, team design, and institutional review process behind an analytic judgment
or forecast — not the judgment's content, but whether the analyst and the shop around them were
set up to reason well. This skill checks whether the reasoning process itself was trained and
coached rather than only the writing, whether competing-view procedures were institutionalized
rather than staged for show, whether management mandated a systematic method on the issues that
warranted it, and whether postmortems and measurement stayed rigorous and unbiased. It audits
the scaffolding around the judgment, never the judgment itself.
When to use
- A judgment or forecast came out of a team, and it is unclear whether that team was actually
psychologically safe, diverse in perspective, and organized around a shared purpose — or
produced groupthink dressed up as consensus.
- A consensus formed after a designated devil's-advocate or similar role weighed in, and it is
unclear whether the agreement is genuine convergence (people who previously held a different
view coming around against their own position or interest) or an artifact of the role itself
manufacturing false confidence that dissent was heard.
- A high-stakes or deception-prone issue was carried on a single-outcome analysis, and whether
management should have mandated — or did mandate — a systematic process such as Analysis of
Competing Hypotheses needs checking.
- A draft is moving through pre-publication review, and the review needs to confirm it actually
interrogated the analyst's unstated assumptions, the alternatives considered and rejected, and
what would change the analyst's mind — ideally by critical thinkers from outside the author's
branch.
- An intelligence failure, a stale estimate, or a suspiciously well-confirmed conclusion needs
diagnosing for whether negative evidence went unweighted or an organizational incentive or
culture problem is at play, rather than defaulting to blaming one analyst's psychology.
- An analyst-development or training plan is under review, to check it targets the analyst's own
reasoning process — coaching, postmortems on both failures and successes, retrospective
evaluation, practice by doing — rather than only writing craft or one-off classroom instruction.
- The analytic process itself is being designed or audited: whether it institutionalizes
competing-view procedures, periodic ground-up re-examination, honest expectation-setting with
consumers about analysis's limits, and blind independent coding wherever results are measured.
Procedure
- Identify which process object is under review — a team's collaborative setup, a group's
convergence on a judgment, a management decision about formalizing method, a
training/development plan, an organizational failure being diagnosed, or a measurement/coding
exercise — before applying the checks below.
- Check the team or group behind the judgment for psychological safety, a shared "we" purpose,
and genuine diversity of perspective rather than ability alone, since these — not raw
individual skill — predict the aggregation gains and open-mindedness a good process should
show (P015).
- If a devil's-advocate or similar role was used, do not credit the resulting consensus at face
value: check whether people who previously held a different view converged independently,
especially against their own prior position or self-interest, and discount any convergence
the role itself could have manufactured by letting the group feel falsely open-minded (P008).
- For a high-stakes or deception-prone issue, check whether management rejected a
single-outcome analysis and mandated a systematic process such as Analysis of Competing
Hypotheses — requiring the alternatives considered, why each was judged less likely, and an
explicit statement that events may not turn out as expected (P010).
- Trace whether the analyst's or team's training and self-examination were directed inward at
their own thinking and reasoning process — not only at organizational procedure, method, or
the substantive topic — and whether that training was coached and learned by doing rather
than delivered as classroom instruction alone (P001, P022, P071).
- When diagnosing an intelligence failure, a stale confirmed estimate, or a lapse in
evidence-weighing, check whether the organization's incentive structure or culture — not only
one analyst's psychology — let negative evidence (the dogs that did not bark) go unweighted,
and whether training and discipline were in place to counter that (P024).
- Check that management supports periodically re-examining key problems from the ground up
rather than relying on incremental adjustment alone, and that consumers were educated about
analysis's limitations as well as its capabilities rather than sold false certainty (P040).
- Confirm the analytic process itself institutionalizes procedures that surface competing
views — analytic debate, devil's advocacy, competitive analysis, peer review, outside
expertise — and rewards weighing multiple hypotheses over accepting the first credible one
(P058).
- For a draft under pre-publication review, confirm the review explicitly questioned the
analyst's mental model — what unstated assumptions underlie the judgment, what alternatives
were considered and why rejected, what would change the analyst's mind — and that reviewers
included capable critical thinkers from outside the author's branch, not only same-branch
colleagues who share the mind-set (P063).
- If a measurement or scoring exercise is in scope, confirm coding was performed by independent
coders blind to the hypotheses, so the analyst's own biases could not leak into the result
(P081).
Inputs
- The analytic judgment, forecast, or draft under review, and whether it was produced by an
individual or a group.
- What is known about the team's setup: size, roles (including any devil's-advocate or similar
role), psychological safety, and diversity of perspective.
- Any record of who held which view before and after a consensus formed, including whether
convergence ran against a person's own prior position or self-interest.
- The issue's stakes and deception risk, and whether a systematic method such as Analysis of
Competing Hypotheses was mandated, considered, or skipped.
- The analyst's or team's training and development history — what it targeted and how it was
delivered — and any record of postmortems or retrospective evaluation.
- The organization's process design: which competing-view procedures are institutionalized, how
consumers are briefed on analysis's limits, and whether periodic ground-up re-examination
happens.
- The pre-publication review record — who reviewed the draft and what was asked — and, if
scoring or coding is in scope, whether coders were independent and blind to the hypotheses.
Output
Per finding: name the process flaw (a devil's-advocate role credited at face value instead of
weighed for genuine independent convergence, an unmandated single-outcome analysis on a
high-stakes or deception-prone issue, training aimed at writing instead of the reasoning process,
a pre-publication review that never questioned the analyst's mental model, non-blind or
non-independent coding, or a competing-view procedure never actually institutionalized), apply
the correction implied by the relevant principle (weigh independent convergence rather than the
role's face value, mandate a systematic method, redirect training and coaching inward and make it
practice-based, bring in outside critical thinkers, use blind independent coders, institutionalize
the missing procedure), state the residual uncertainty the correction leaves, and end with a
concrete next step. Order findings highest-impact first. Never substitute a bare pass/fail
verdict for this structure, and never produce the substantive analytic judgment yourself.
Anti-patterns to flag
- A devil's-advocate or similar role credited as proof the group considered dissent, with the
false confidence of having been open-minded substituted for evidence of genuine independent
convergence (P008).
- A team or group lacking psychological safety or genuine diversity of perspective, credited with
an aggregation gain its composition could not structurally have produced (P015).
- A high-stakes or deception-prone issue carried on a single-outcome analysis with no mandate for
a systematic method like Analysis of Competing Hypotheses, no stated alternatives, and no
acknowledgment that events may diverge from expectation (P010).
- Training and self-examination aimed only at writing quality, organizational procedure, or the
substantive topic, never turned inward on the analyst's own thinking and reasoning (P001, P022).
- Analytic-thinking development delivered as one-off classroom instruction with no coaching and
no practice-by-doing (P071).
- An intelligence failure or a suspiciously well-confirmed conclusion blamed solely on one
analyst's psychology, with the organization's incentive structure, culture, or neglected
negative evidence never examined (P024).
- Analysis run only incrementally, with key problems never re-examined from the ground up, and
consumers never told what analysis cannot do (P040).
- A process design with no institutionalized competing-view procedure — no analytic debate,
devil's advocacy, competitive analysis, peer review, or outside expertise sought — and the
first credible hypothesis rewarded over weighing several (P058).
- Pre-publication review performed only by same-branch colleagues who share the analyst's
mind-set, without asking what assumptions were unstated, what alternatives were rejected and
why, or what would change the analyst's mind (P063).
- Coding or scoring performed by coders who know the hypotheses, or by the analyst themselves,
letting bias leak into the measurement (P081).
References
See ../../references/analytic-method-principles-index.md for the full principle catalogue. For
adjacent concerns, see the sibling skills: competing-hypotheses-and-diagnostic-evidence runs
the Analysis of Competing Hypotheses matrix itself — diagnosticity weighing and
refutation-seeking — once this skill's review confirms formalization was warranted;
structured-analytic-techniques selects and sequences which technique to apply;
assumptions-framing-and-analytic-writing reviews an individual draft's explicit linchpin
assumptions and its separation of fact from opinion; cognitive-biases-and-dual-process-reasoning
covers the individual analyst's bias catalogue that the team, training, and review design in
this skill exist to counter.
Provenance
Derived solely from P001, P008, P010, P015, P022, P024, P040, P058, P063, P071, and P081 (this
package's six analytic-tradecraft sources: Psychology of Intelligence Analysis; Thinking, Fast
and Slow; Superforecasting; Expert Political Judgment: How Good Is It? How Can We Know?;
Perception and Misperception in International Politics; and the CIA Sherman Kent School's
Tradecraft Primer — all distillation-only; see the frontmatter above for the full claim list).
