analyze

name

analyze

description

Rigorous, skeptical statistics on result data — the anti-p-hacking workflow between do (experiments make data) and write (claims about data). Point /analyze at a data file (CSV/TSV/parquet/json) or a results directory (e.g. from /investigate) and the question; it profiles the real data, picks the CORRECT test for THIS data shape and checks its assumptions, reports effect sizes WITH confidence intervals (never a p-value alone), corrects for multiple comparisons (or says why none is needed), fits/compares models, makes B&W publication figures, and runs an adversarial threats-to-validity pass — then renders one fixed-template HTML analysis in the atlas (~/.anu/atlas/analyses/<slug>). Use when the user runs /analyze or asks to analyze / run statistics on / find the effect in / test for significance in / check the assumptions of / make figures from result data, or asks whether a result is real or over-claimed.

/analyze — honest statistics on your numbers

Between do (experiments produce data) and write (claims about data) is where most papers fail: p-hacking, no effect sizes, no confidence intervals, the wrong test, no assumption checks, no multiple-comparison correction. /analyze is the discipline that closes that gap — be your own skeptic about your numbers. Profile the data, pick the correct test (and check its assumptions), report effect sizes and CIs (never p alone), correct for multiple comparisons, fit and compare models, make publication figures, and run an adversarial threats-to-validity pass. The page is the artifact — keep the chat output minimal.

The fourth bend of the research arc — find (/prior-art) → understand (/study) → do (/investigate) → analyze (this) → show (/present). It reads what /investigate produced and hands /present numbers that survive scrutiny.

Steps

1. Resolve the data and the question (one bash call). $ARGUMENTS is a data file (CSV/TSV/parquet/json) or a results directory, plus the question being asked of the data. If the question is unstated, infer candidates from the columns and confirm with the user before spending the workflow — an analysis of the wrong question is worse than none. Profile the REAL data once so the plan is grounded in it, not in the hope:

   DATA="<abs path to the file or dir>"
   STAMP=$(date '+%Y-%m-%d %H:%M'); TS=$(date +%Y%m%d-%H%M%S)
   PROFILE=$(python3 - "$DATA" <<'PY'
   import sys, pathlib, pandas as pd
   p = pathlib.Path(sys.argv[1])
   f = p if p.is_file() else next((q for q in sorted(p.rglob('*')) if q.suffix in
        ('.csv','.tsv','.parquet','.json')), None)
   if f is None: print('no tabular file found under', p); sys.exit(0)
   df = (pd.read_parquet(f) if f.suffix=='.parquet' else
         pd.read_json(f) if f.suffix=='.json' else
         pd.read_csv(f, sep='\t' if f.suffix=='.tsv' else ','))
   print('file:', f); print('shape:', df.shape)
   print('dtypes:\n', df.dtypes.to_string())
   print('missing:\n', (df.isna().mean().round(3)).to_string())
   print('head:\n', df.head().to_string())
   PY
   )
   SLUG=$(printf '%s' "<the question>" | tr '[:upper:]' '[:lower:]' \
          | tr -cs 'a-z0-9' '-' | sed 's/^-//;s/-$//' | cut -c1-60)
   OUT="$HOME/.anu/atlas/analyses/$SLUG"; mkdir -p "$OUT/fig" "$OUT/history"
   

   If the profile reports it couldn't load the data, stop and resolve the path before running the workflow.

2. Run the workflow. Call the Workflow tool with:

   • scriptPath: ~/.local/share/anu/plugins/analyze/skills/analyze/workflow.js (expand ~ to the real home dir)
   • args: {"question": "<the question>", "data_path": "<abs path>", "out_dir": "<OUT>", "profile": "<the PROFILE text>", "generatedAt": STAMP}

   The workflow profiles → plans → runs → skeptic → synthesizes. It must run the analysis for real — preferring the axiomatic MCP model-fitters when the question is a fit/comparison (reached via ToolSearch: AxModelFitter_fit_model / compare_models / cross_validate_model / calculate_information_criteria; AxModelFitterV2_generate_code + execute_code), otherwise python (numpy/scipy/statsmodels/pandas/matplotlib) in a contained box — and it WRITES figures into <out_dir>/fig/*.png, returning their RELATIVE paths ("fig/x.png"). It returns the analysis JSON.

3. Persist and render (bash + Write). Move any prior analysis to history, write the returned JSON, render, open — figures resolve relative to index.html:

   [ -f "$OUT/analysis.json" ] && mv "$OUT/analysis.json" "$OUT/history/$TS.json"
   

   Write the returned JSON to $OUT/analysis.json, then:

   python3 ~/.local/share/anu/plugins/analyze/skills/analyze/render.py "$OUT/analysis.json" "$OUT/index.html"
   open "$OUT/index.html"
   

4. Report (minimal): the headline stated honestly — the effect size and its CI, not a bare p-value — the single strongest threat-to-validity, and the figures path. Do not restate the page; the page is where it lives.

Rules — the anti-p-hacking contract

• Never report a p-value without an effect size AND a confidence interval. A p alone is the slop this workflow exists to prevent.
• Always state the multiple-comparison correction (Holm / BH / Bonferroni), or that none was needed and why — decided in the plan, before seeing p-values.
• Pick the test for the data shape, not by reflex — check its assumptions and fall back to the robust/nonparametric alternative when one breaks.
• The analysis must RUN, reproducibly. Axiomatic MCP fitters when available, else python in a contained box. Which path ran is recorded in code_note.
• Claims stay bounded — "estimate X, 95% CI [a, b], effect d=…" — never a blanket "significant" / "novel" / "proven" / "SOTA". If the CI crosses the null or a threat is fatal, the headline says so plainly.
• Never edit template.html / render.py per run — fixed by design; all per-run content (including the figure paths) lives in analysis.json. Same contract as /map, /study, /investigate.
• History accumulates in $OUT/history/; never delete it — it's how an analysis is diffed as the data or the question moves.