<div align="right">
**English** · [中文](./README.zh-CN.md)
</div>
<br />
<div align="center">
# meta-analysis-skill
**A publication-grade meta-analysis & systematic review, run by your coding agent — with you signing off at the points that matter.**
<br/>
Built natively for Claude Code; runs in Codex and any agent that loads the SKILL.md format.
<br/>
Five skills · seven human sign-off gates · red lights computed by R, not the LLM.
<br/>
<a href="LICENSE"><img src="https://img.shields.io/badge/license-PolyForm--NC%20%2B%20CC%20BY--NC--SA-000?style=flat-square" alt="Dual-licensed: PolyForm-NC (code) + CC BY-NC-SA (docs)"/></a>
<img src="https://img.shields.io/badge/source--available-non--commercial-000?style=flat-square" alt="source-available, non-commercial"/>
<img src="https://img.shields.io/badge/agent-Skill-000?style=flat-square" alt="Agent Skill"/>
<img src="https://img.shields.io/badge/R-metafor-000?style=flat-square" alt="R · metafor"/>
<img src="https://img.shields.io/badge/Python-3.10+-000?style=flat-square" alt="Python 3.10+"/>
<img src="https://img.shields.io/badge/standards-PRISMA_2020_·_GRADE_·_AMSTAR_2-000?style=flat-square" alt="Standards"/>
<img src="https://img.shields.io/badge/version-0.2.0-000?style=flat-square" alt="v0.2.0"/>
</div>
<br/>
<br/>
## What it is
You ask your agent for a meta-analysis or a systematic review. This runs the **real** pipeline — protocol, PRISMA-S search ingestion, dual-blind screening with Cohen's κ, dual extraction, **statistics executed locally in R** (`metafor` and friends), forest/funnel figures, and an APA / PRISMA-2020 draft — and stops at **seven irreversible points** to hand you an audit card built for that decision. You read the card, sign off or send it back.
The agent does the grunt work that used to eat hundreds of research-assistant hours. You make the calls that can't be delegated — and that carry authorship. This is **not** a one-click paper machine, and it does not pretend to be (see [Honest boundaries](#honest-boundaries)).
```text
In your agent's chat, after install:
Do a three-level meta-analysis of empathy on emotion recognition,
with intervention type as the key moderator.
```
<br/>
## What makes it different
Three design decisions separate this from "ask the model to do a meta-analysis."
**1 · Red lights are computed by R, not read by the LLM.**
`audit_parse.R` derives every statistical flag — I² > 90%, a single study carrying > 30% of the weight, PET intercept non-significant, funnel asymmetry — directly from the fitted `metafor` object's fields. The model never eyeballs a `summary()` printout to decide what's significant. If the engine says red, it's red.
**2 · The seven sign-off gates actually block.**
Each gate's verdict lands in a `decisions/` file. Before the next stage runs, `signoff_preflight.py` reads it — **unsigned, rejected, or failing a hard check → non-zero exit code, and the downstream stage cannot start.** It is not a prompt nudge to "please review." Hard checks (a red light, κ below threshold, an unverified citation) are evaluated *before* the human sign-off and block even a signed gate — so a rubber-stamp can't wave through a broken result.
**3 · Gold-standard calibration answers "how accurate is the AI?" with data.**
Hand-code 10–20 studies. The agent extracts the same ones with its real production prompt. `compute_kappa_icc.R` scores per-field agreement (Cohen's κ for categorical fields, ICC for continuous) and sorts every field into **green (trust it, skip review) · amber (spot-check) · red (review every one)**. That per-field table is your evidence-based answer to *which fields you can trust the agent on* — not a vibe.
<br/>
## What's in the box
Five interlocking skills. `meta-analysis` and `systematic-review` consume the shared SSOT base by reference; `academic-ref-check` guards the final citation gate; `manuscript-typeset` renders the finished manuscript for submission.
| Skill | What it owns | Stage |
|---|---|---|
| **systematic-review** | Strict PRISMA 2020 / Cochrane v6.5: protocol, PRISMA-S search, dual-blind screening + κ, dual extraction, RoB 2 / ROBINS-I, GRADE, AMSTAR 2 | L1–L8 |
| **meta-analysis** | Quantitative synthesis: effect sizes, FE/RE, I²/τ²/95% PI, subgroup & meta-regression, publication bias, psychology methods (RVE · three-level · MASEM · Hunter-Schmidt) | L9–L12 |
| **review-methodology-foundations** | The shared base — **20 Single-Source-of-Truth** methodology documents the two review skills cite (effect-size decisions, synthesis methods, RoB tool mapping, GRADE, AMSTAR 2, reporting standards, AI-failure defenses, …) | all (referenced) |
| **academic-ref-check** | End-to-end reference checking, APA 7 formatting, HTTP verification against OpenAlex + CrossRef — drives the final citation gate | L11 |
| **manuscript-typeset** | Standalone academic typesetting: a finished Markdown manuscript → submission-ready `.docx` + APA 7 (`man`/`jou`) PDF + a clean Markdown copy, behind a **fidelity gate** that fails the build if any number in the output has no source in the input | L11–L12 |
<br/>
## The supervised pipeline
Twelve methodology phases (L1–L12). The agent runs them; you stand at seven irreversible gates.
```
L1–L3 type · PICO · protocol ── you: define the question, confirm scope
L4 search strategy ───────────────────▶ SP1 freeze search + eligibility
L5 ingest exports · dedup (PRISMA-S)
L6 dual-blind screening + κ ──────────▶ SP2 resolve disagreements
L7 calibrate ────────────────────────▶ SP3 approve extraction go-live
dual extraction + numeric check ──▶ SP4 verify key numbers
L8 risk of bias (RoB 2 / ROBINS-I)
L9 EFFECT SIZES → MODEL → I²/τ²/PI ──▶ SP5 accept model + heterogeneity
moderators · sensitivity · pub-bias▶ SP6 accept interpretation
L10 GRADE
L11 APA / PRISMA draft + citations ────▶ SP7 clear flagged citations
L12 AMSTAR 2 self-assessment
↑ AI does the grunt work ↑ seven gates a script enforces
```
You also do what no agent can: run the database searches that have no API (export RIS/CSV from Web of Science, PsycINFO, CNKI), fetch paywalled PDFs, write the introduction and discussion, and put your name on it. Realistic human effort drops from hundreds of hours to roughly **5–10 working days of judgment** concentrated at the gates.
<br/>
## Cost & effort
Running this for real is **not free and not instant** — supervised, publication-grade evidence synthesis has an irreducible floor. Three separate costs, kept separate so you can plan each:
**1 · Compute / model API.** Order of magnitude **tens to a few hundred USD** for one full review, depending on how you pay for the model — a flat Max/Pro subscription absorbs most of it; metered API billing sits at the higher end. The R statistics, figure rendering, and preflight run locally and cost nothing.
**2 · Agent wall-clock — roughly 10–15 hours** for a full run. Most of it is **unattended** — you leave it running: search ingestion and dedup, batched dual-blind screening, dual extraction, R pooling, figure rendering, drafting. The **attended** part is the seven sign-off gates — you have to be at the keyboard to read each audit card and decide.
**3 · Human labour — a few hours, concentrated where it can't be delegated:**
| Human task | Rough effort |
|---|---|
| Run the no-API database searches, export RIS/CSV | 2–4 h |
| Fetch paywalled full text | variable — depends on your access |
| The seven sign-off gates | ~10–30 min each → **~1.5–3.5 h total** |
| Write the introduction & discussion, put your name on it | separate scholarly work — hours to days |
**A rough scaling handle.** The single real run this was validated on screened **23,626 records** (of which **4,715** went through dual-blind screening + adjudication), pulled **~1.5 GB** of full text, and used **~97 SubAgent dispatches** end to end over ~10–15 h. As a blended, whole-pipeline rate that is **≈ 4 SubAgent dispatches per 1,000 records screened** — a crude single-run estimate, **not a formula**. It moves a lot with the number of databases, *k* (studies that survive), and how many records survive dedup; treat it as an order of magnitude, not a quote.
**Why "fast" has a floor.** The wall-clock lower bound comes from the **methodology**, not from slow engineering: dual-blind screening, dual extraction, and seven sign-off gates are structural. Wall-clock does **not** scale linearly with record count — a review with 10× fewer records is not 10× faster, because the fixed human-gate loop and the longest-pole stages dominate. The only honest ways to spend less are **narrower scope** (smaller *k*, fewer databases) or an explicit **downgrade** (Learning mode, every shortcut disclosed) — never skipping a gate.
<br/>
## The audit cards
Every gate renders a self-contained, offline, bilingual HTML card — numbers from the engine, red/amber/green from R, plain-language explanation of *what to look at and what you're deciding*. Open one in any browser.
<table>
<tr>
<td width="50%" valign="top"><img src="assets/SP1_search_freeze.png" alt="SP1 search freeze"/><br/><sub><strong>SP1 · Freeze search</strong> — seed-recall check before the search is locked.</sub></td>
<td width="50%" valign="top"><img src="assets/SP2_screening_caution.png" alt="SP2 screening"/><br/><sub><strong>SP2 · Screening</strong> — Cohen's κ gauge; only disagreements surface for you.</sub></td>
</tr>
<tr>
<td valign="top"><img src="assets/SP3_calibration_pass.png" alt="SP3 calibration"/><br/><sub><strong>SP3 · Calibration</strong> — per-field reliability table; which fields go live unreviewed.</sub></td>
<td valign="top"><img src="assets/SP4_numeric_verify.png" alt="SP4 numeric verify"/><br/><sub><strong>SP4 · Numeric check</strong> — every effect-size input checked against the source figure.</sub></td>
</tr>
<tr>
<td valign="top"><img src="assets/claims_vs_data.png" alt="SP6 claims vs data"/><br/><sub><strong>SP6 · Claims vs data</strong> — every claim in the draft checked against what the analysis actually supports.</sub></td>
<td valign="top"><img src="assets/SP7_citation_verify.png" alt="SP7 citation"/><br/><sub><strong>SP7 · Citations</strong> — HTTP-verified references; suspected fabrications flagged red.</sub></td>
</tr>
</table>
<br/>
## Install
The five skills are interdependent — install all five into your agent's skills directory. Three commands: install dependencies, symlink the skills, health-check.
```bash
git clone https://github.com/O0000-code/meta-analysis-skill.git
cd meta-analysis-skill
./setup.sh # install dependencies: R packages + node_modules + Python (one command)
./install.sh # symlink the five skills into your agent (default: ~/.claude/skills)
python3 doctor.py # health check — what's present/missing and what each gap costs
# ./install.sh ~/.codex/skills # Codex CLI
# ./install.sh ~/.agents/skills # cross-agent convention
```
`setup.sh` installs what it can without sudo — the R/CRAN packages (`metafor`, `robumeta`, `clubSandwich`, `metaSEM`, `psychmeta`, `puniform`, `irr`, `psych`, `jsonlite`, …) plus `dmetar` from GitHub, the audit-card renderer's `node_modules`, and the Python figure/docx dependencies (`matplotlib`, `numpy`, `python-docx`, `pyyaml`). It only *prints* the command for things that need sudo or a large download — `pandoc` and XeLaTeX/TinyTeX (for `manuscript-typeset`), `JAGS` (for `RoBMA`), and CJK/serif fonts.
`doctor.py` then reports exactly what is present or missing and, for each gap, **what capability you lose** — so a missing optional dependency degrades a feature loudly instead of failing silently. When `JAGS` is absent, the pipeline auto-falls back to `puni_star` + PET-PEESE and says so. The Python helpers (`verify_http.py`, `signoff_preflight.py`, ingestion) use only the standard library.
> **Companion skills** (search, figure styling, PDF fetch, full-text conversion) are optional and shipped separately — where to get each, and how the pipeline degrades without it, is in [`docs/DEPENDENCY-MATRIX.md`](docs/DEPENDENCY-MATRIX.md).
<br/>
## Try it in 2 minutes
Want to see the outputs before committing to a real run? The bundled demo runs the **L9 → L12 tail** on a **synthetic 10-study dataset** — one command, ~1–2 minutes of compute:
```bash
bash demo/run_demo.sh
```
It produces the full shape of a finished run: a **forest** plot, a **funnel** plot, the **PRISMA 2020** flow diagram, and a **risk-of-bias** traffic-light figure; two **sign-off cards** (SP5 model + heterogeneity, SP6 publication bias); and a **mini delivery manifest**. Open the HTML cards in any browser — they are fully offline. The figures come from the skill's fixed renderers (each with its own QC gate), not ad-hoc plotting code.
> **Synthetic data, not a real search.** The 10 studies, effect sizes, PRISMA counts, and RoB ratings are all invented (`Demo-Study-NN` placeholders) — the demo shows the *artefact forms*, not a result. What a real, human-supervised run actually costs is in [Cost & effort](#cost--effort) above. See [`demo/README.md`](demo/README.md) for details.
<br/>
## Three modes
Set in L3; the same methodology, three human-involvement profiles.
| Mode | For | Human role |
|---|---|---|
| **Strict** | Top-journal submission, the fully-automated baseline | Final acceptance, once |
| **Learning** | Coursework, proposals — simplified, but every shortcut disclosed | Sampled spot-checks |
| **Supervised** | Publishable output where you sign off at every irreversible point | **Decision review at all seven gates** |
Supervised mode is the headline: it carries the **full strict-mode methodology** (nothing is downgraded) and adds the seven structural gates on top. It's what you want when "AI drafts it, a human checks every critical call, and it has to reach publication grade" describes the job.
<br/>
## Honest boundaries
> **Responsible use.** AI drafts, a human signs off at seven irreversible gates, and the signing author writes the theory and takes full responsibility for every claim — fully-unsupervised auto-submission is out of scope by design, not omission. The full stance, the two-phase supervision model, and the honest boundaries are in [`RESPONSIBLE_USE.md`](RESPONSIBLE_USE.md).
This project's standing rule is to never claim more certainty than the evidence supports — that applies to the tool itself:
- **Not a one-click paper.** Search execution (databases without an API need manual export), paywalled full text, the seven sign-off judgments, the introduction and discussion, and authorship are human work by design. The end state is *AI does the grunt work + a human signs off at the gates + a human writes the theory + a human takes responsibility.*
- **Fully-autonomous "no human" mode is out of scope** — for technical and accountability reasons, not as a missing feature.
- **Mechanism validated on synthetic data; the real-data "last mile" is in progress.** The supervised engine (gates, calibration, red-light computation) has been exercised end-to-end on a synthetic pilot. Validating the full chain on **real PDFs and real searches** is ongoing; the synthetic pilot's accuracy numbers are an optimistic upper bound, not real-world accuracy.
<br/>
## Methodology & attribution
The 20 SSOTs operationalize methodology from the Cochrane Handbook v6.5, Cooper, Borenstein, the JBI Manual, and the APA Publication Manual, and reference the PRISMA 2020, GRADE, AMSTAR 2, and APA JARS standards — each with citations. Statistical methods, formulas, and thresholds are facts and procedures (not subject to copyright); they are operationalized in our own words with attribution, never reproduced as verbatim prose. Standardized instruments (RoB 2, ROBINS-I, AMSTAR 2) are quoted with their official sources because an instrument must keep its exact wording. Full credits — sources, instruments, bundled runtime assets, and dependencies — are in [`THIRD_PARTY.md`](THIRD_PARTY.md).
<br/>
## License
**Source-available, non-commercial — dual-licensed by file type.** Code (`scripts/**`, `*.py` / `*.R` / `*.mjs` / `*.sh`, `setup.sh` / `install.sh` / `doctor.py`) is under [PolyForm Noncommercial 1.0.0](LICENSES/PolyForm-Noncommercial-1.0.0.txt); documentation (`*.md`, `SKILL.md`, `references/**`, `assets/**`) is under [CC BY-NC-SA 4.0](LICENSES/CC-BY-NC-SA-4.0.txt). [`LICENSE`](LICENSE) holds the file-type split; [`LICENSES/`](LICENSES) holds the full texts.
This is **source-available, not OSI open source** — PolyForm Noncommercial permits use, modification, and sharing for non-commercial purposes, and CC BY-NC-SA lets you fork and improve the docs under the same terms. Commercial licensing is available by separate agreement (contact the copyright holder). GitHub's sidebar shows **"Other"** because a genuine dual-licensed repo has no single-license badge — that is the correct, honest result, not a misconfiguration. This license applies going forward; earlier versions released under MIT keep those terms (rights already granted are not revoked).
The methodology sources and standardized instruments it cites (RoB 2, PRISMA 2020, GRADE, AMSTAR 2, …) carry their own terms — see [`THIRD_PARTY.md`](THIRD_PARTY.md).