| name | cogpsych-data-analysis |
| description | Use when analyzing data and fitting/comparing models for a Cognitive Psychology (Elsevier) manuscript. The journal expects principled model fitting and comparison (AIC/BIC/Bayes factors), parameter and model recovery, (generalized) linear mixed models or hierarchical Bayesian estimation where apt, and effect sizes with uncertainty — all reproducible from shared code. Guides the analysis and modeling; it does not fabricate results. |
Data Analysis & Model Fitting (cogpsych-data-analysis)
Cognitive Psychology holds analyses to a model-based standard: fit the formal model, compare it
to rivals with principled criteria, demonstrate that parameters and models are recoverable, use
mixed models or hierarchical Bayesian estimation where the design demands it, and report effect
sizes with uncertainty for behavioral results — all regenerable from deposited code. This is the
experiment-to-model-fit loop that defines the venue.
When to trigger
- Fitting the formal model and comparing it to rival accounts
- Running the behavioral analyses (mixed models, hierarchical Bayesian, contrasts)
- A reviewer asked for model comparison, recovery, robustness, or fuller disclosure
- Preparing analysis/model code and a data dictionary for deposit
Reporting norms Cognitive Psychology expects
- Fit and compare models, don't just fit one. Report fit for your model and the rival(s) under
matched flexibility; compare with AIC/BIC, cross-validation, or Bayes factors as
appropriate, and say what the comparison licenses.
- Show recovery. Demonstrate parameter recovery (can the fitting procedure recover known
parameters from simulated data) and model recovery (does the comparison criterion pick the
generating model) — without these, a fit edge is not interpretable.
- Use the right hierarchical structure. Crossed random effects over subjects and items call for
(generalized) linear mixed models; for cognitive models, hierarchical Bayesian estimation
pools strength across participants. Justify the structure; don't aggregate away the variance.
- Effect sizes + uncertainty for behavior. Report standardized/unstandardized effect sizes with
confidence/credible intervals for key behavioral results, not just p-values and stars.
- Confirmatory vs. exploratory. Separate pre-committed model comparisons and tests from
exploratory model exploration; do not present a post hoc winning model as predicted.
- Reproducible. Model and analysis code, with seeds and pinned versions, regenerate every reported
fit, figure, and table in a fresh session (see
cogpsych-open-science-and-transparency).
Robustness
- Show the conclusion survives reasonable alternative model specifications, priors (for Bayesian fits),
and exclusion choices; report sensitivity, not a single fragile fit. Report convergence diagnostics
(e.g., R-hat, ESS) for Bayesian models.
Worked micro-example (illustrative numbers)
A preregistered three-experiment recognition-memory program fitting UVSD vs. DPSD to confidence-ROC data.
Model comparison (preregistered) — pooled across Exps 1-3
Fit (hierarchical Bayesian, matched flexibility):
UVSD favored: dBIC = 14 vs. DPSD; Bayes factor ~ 30 in favor of UVSD
Recovery (required): parameter recovery good (recovered d', sigma within
credible intervals); model recovery ~ 92% correct at the design's N/trials
Diagnostic signature: z-ROC slope 0.78, 95% CrI [0.72, 0.84], and linear
(no reliable curvature) — the qualitative pattern UVSD predicts and DPSD
forbids, consistent across all three experiments
Behavioral effect (mixed model)
List-strength manipulation on d': b = 0.31, 95% CI [0.18, 0.44]
Exploratory (labeled)
A small response-bias drift surfaced post hoc; reported as exploratory
Why this passes Cognitive Psychology scrutiny: the model is compared (not just fit), recovery
makes the comparison interpretable, the qualitative signature corroborates the fit index, hierarchy
respects subject/item variance, and the exploratory drift is honestly demoted.
Analysis-stage reviewer pushback and the venue fix
| Reviewer pushback | What it signals here | Cognitive Psychology fix |
|---|
| "You only fit your model" | one-model storytelling | fit the rival under matched flexibility; report AIC/BIC/BF and what it licenses |
| "Better fit may be overfitting" | flexibility imbalance | add model recovery + cross-validation; penalize complexity |
| "Can you recover these parameters?" | identifiability doubt | run and report parameter + model recovery simulations |
| "Aggregated means hide variance" | wrong error structure | refit with crossed-random-effects mixed model / hierarchical Bayesian |
| "Is this the model you predicted?" | post hoc selection | pre-commit the comparison; relabel post hoc fits exploratory |
| "I can't rerun your fits" | reproducibility gate | ship seeded model code + a fresh-session run log |
Calibration anchors
- A model that is fit, compared, and recovered is the unit of evidence here — a single fit with a
good index but no rival and no recovery is not persuasive.
- Trust a crossed qualitative prediction over a marginal fit advantage; report both and lead with
the signature the rival forbids.
- Respect the data's hierarchy: aggregating over subjects or items inflates false positives and can
bias parameter estimates; use mixed/hierarchical models and justify the random-effects structure.
- For Bayesian fits, report priors, convergence, and sensitivity — a fit without diagnostics is not
reproducible evidence.
Execution bridge (StatsPAI / Stata MCP)
Run the battery, don't just enumerate it. Full map:
execution-with-mcp. Cognitive Psychology is experimental — within-subject designs and mixed models dominate; report the model, the effect size, and multiple-comparison control.
- Many outcomes / specifications:
romano_wolf (step-down FWER) or
benjamini_hochberg — report the adjusted threshold.
- OVB sensitivity:
oster_delta / sensemakr.
- Inference:
wild_cluster_bootstrap (few clusters), twoway_cluster / conley;
multilevel data → cluster at the right level.
- Re-fit off one handle:
audit_result(result_id) lists the missing checks and the
exact suggest_function for each.
- Exhibits:
etable / did_summary_to_latex from the handle — no retyped numbers.
Keep the decisive checks in the body and the exhaustive battery in the supplement. See
the executed chain in the JF execution walkthrough.
Anti-patterns
- Fitting only your model with no rival and no comparison criterion
- Claiming a fit advantage without matched flexibility, recovery, or cross-validation
- Aggregating to cell means and ignoring crossed subject/item variance
- p-values and stars with no effect size or interval for behavioral results
- Presenting a post hoc winning model as a predicted result
- Model/analysis code that does not regenerate the reported fits
Output format
【Model comparison】rivals fit under matched flexibility + criterion (AIC/BIC/BF)? [Y/N]
【Recovery】parameter + model recovery reported? [Y/N]
【Hierarchy】mixed model / hierarchical Bayesian where apt + diagnostics? [Y/N]
【Behavioral effects】effect sizes + intervals? [Y/N]
【Confirmatory vs exploratory】separated? [Y/N]
【Reproducible】seeded code + data dictionary + fresh-session check? [Y/N]
【Next】cogpsych-tables-figures
Supplementary resources
