stata-skill-contributor

name	stata-skill-contributor
description	Guide for contributing to the stata-skill project. Use when the user wants to run the eval pipeline, analyze test results, improve reference docs, add new package documentation, or work on roadmap items. Covers the testing infrastructure, multi-agent analysis workflow, cost estimates, and links to prior eval results and improvement history.

Stata Skill Contributor Guide

This skill helps you contribute to the stata-skill project — the Stata reference skill for Claude Code. It covers how to run the evaluation pipeline, analyze results, and make data-driven improvements to the skill documentation.

Cost Warning

Running the eval pipeline uses Claude Agent SDK query() calls, which consume API credits.

What	Approximate cost
Single task, 1 run	$0.50 - $1.50
Single task, 5 runs	$2.50 - $7.50
All 24 tasks, 1 run each	$12 - $36
All 24 tasks, 5 runs each	$60 - $120
Full A/B comparison (2 arms × 24 tasks × 5 runs)	$120 - $240
Multi-agent post-test analysis (5 Sonnet subagents)	$3 - $8

Start small: run 1-3 tasks relevant to your change before running the full suite. Use --runs 1 to validate the pipeline works before scaling up.

Quick Reference

File	What it is
tests/eval.py	Evaluation harness — sends tasks to Claude, judges responses
tests/rubric.md	7-category scoring rubric (max 55 points)
tests/tasks/task_*.md	24 test task definitions
tests/runner-only-plugin/	Minimal baseline plugin (execution instructions only)
tests/results/	All run directories and aggregate JSON results
tests/EVAL_RESULTS_2026-03-22.md	Prior eval results with full analysis and run index
ROADMAP.md	Open problems and improvement directions
CONTRIBUTING.md	General contribution guide (adding packages, filing issues)

Running the Eval Pipeline

Basic commands

# Single task, single run — fast sanity check (~$1)
python tests/eval.py tests/tasks/task_01_data_cleaning.md

# Single task, 5 runs — measure variance (~$5)
python tests/eval.py tests/tasks/task_01_data_cleaning.md --runs 5

# Save results as a named baseline
python tests/eval.py tests/tasks/task_01_data_cleaning.md --runs 5 \
    --save tests/results/my_baseline.json

# Compare against a baseline
python tests/eval.py tests/tasks/task_01_data_cleaning.md --runs 5 \
    --compare tests/results/my_baseline.json

A/B testing a doc change

# 1. Save baseline BEFORE your edit
python tests/eval.py tests/tasks/task_16_multiple_imputation.md --runs 5 \
    --save tests/results/before_mi_fix.json

# 2. Make your edit to the reference doc

# 3. Re-run and compare
python tests/eval.py tests/tasks/task_16_multiple_imputation.md --runs 5 \
    --compare tests/results/before_mi_fix.json

Look for: mean score going up, SD staying flat or going down, no new failure modes.

Testing modes

# Full skill (default) — loads the complete plugin
python tests/eval.py tests/tasks/task_*.md

# Runner-only — minimal plugin with just execution instructions, no reference docs
python tests/eval.py tests/tasks/task_*.md --runner-only

# No skill — no plugin at all
python tests/eval.py tests/tasks/task_*.md --no-skill

# Alternative skill version — point at a different plugin directory
python tests/eval.py tests/tasks/task_*.md --skill-path /path/to/modified/plugin

The runner-only baseline is the meaningful comparison. It isolates what the reference material adds beyond "here's how to run Stata." The no-skill baseline is weaker (agent can't even execute code).

Parallel execution

Run multiple tasks in parallel by launching separate processes:

python tests/eval.py tests/tasks/task_01_data_cleaning.md --runs 5 \
    --save tests/results/v3_task01.json &
python tests/eval.py tests/tasks/task_07_did.md --runs 5 \
    --save tests/results/v3_task07.json &
wait

Batch 3-4 tasks at a time to avoid resource exhaustion. Run directories are created atomically, so parallel execution is safe.

Output structure

Each run creates tests/results/run_NNN/ containing:

File	Contents
transcript.json	The test agent's full response
judge_findings.md	Per-category scores with justifications
metadata.json	Score, model, skill_mode, cost, duration, timestamp
code_files/	Any .do/.log files the agent wrote

Aggregate results are saved to the --save path as a JSON list of per-run results.

Multi-Agent Post-Test Analysis

After running evals, dispatch Sonnet subagents to analyze the results in parallel. This is the workflow that surfaces actionable improvements.

Step 1: Generate the comparison table

# Quick script to build the comparison table from saved results
import json, statistics
for i in range(1, 25):
    tn = f"{i:02d}"
    with open(f"tests/results/v3_task{tn}.json") as f:
        full = [r["score"] for r in json.load(f) if r["score"] is not None]
    with open(f"tests/results/v3_runner_task{tn}.json") as f:
        runner = [r["score"] for r in json.load(f) if r["score"] is not None]
    fm, rm = statistics.mean(full), statistics.mean(runner)
    print(f"Task {tn}: full={fm:.1f} runner={rm:.1f} delta={fm-rm:+.1f}")

Step 2: Dispatch analysis subagents

Launch 4-5 parallel subagents, each investigating a specific pattern in the results. The prompts below are templates — adapt them to your actual results.

Agent 1 — Regression analysis (skill hurts):

Investigate tasks where the full skill scores LOWER than runner-only. Read the result JSONs, the relevant reference docs, and the judge reasoning for low-scoring runs. Identify specific examples in the reference docs that teach wrong patterns. Compare what the with-skill agent does vs the runner-only agent.

Agent 2 — Redundancy analysis (skill adds nothing):

Investigate tasks where full skill and runner-only score identically. Read the reference docs that would be triggered. Determine whether the content is redundant — the model already knows this material from training. Identify which reference files could be deprioritized or trimmed.

Agent 3 — High-variance analysis (unreliable tasks):

Investigate tasks with SD > 10. Compare the LOW-scoring run vs HIGH-scoring run for each. Identify what went wrong — wrong syntax in the doc? Agent misinterpretation? Context collapse (agent wrote code for wrong task)? Quote specific code that failed.

Agent 4 — Biggest wins analysis (what works):

Investigate the 5 tasks with the largest positive deltas. Read the runner-only judge reasoning to identify specific knowledge gaps. Categorize: community package syntax, Stata-specific idioms, workflow patterns, gotcha avoidance.

Agent 5 — Graphics/weak area analysis:

Investigate the lowest-scoring full-skill tasks. Compare successful vs failed runs. Identify missing content in the reference docs.

Step 3: Compile findings and make changes

Each agent returns specific, evidence-backed proposals. Typical findings:

• Bugs in docs: Wrong syntax that the agent copies faithfully (highest priority to fix)
• Missing gotchas: Common errors the docs don't warn about
• Missing workflows: Multi-step patterns the agent needs but can't find
• Redundant content: Topics the model already knows (lowest priority)

Fix bugs first, add missing gotchas second, then re-run to verify improvement.

Example: prior results

See tests/EVAL_RESULTS_2026-03-22.md for a complete example of this workflow applied to the full 24-task suite. That analysis:

• Found 4 bugs in reference docs (wrong mi estimate pattern, wrong ssc install command, wrong variable names)
• Added 3 missing gotchas (graph box/twoway incompatibility, DiD event-study syntax, nlcom clobbers e())
• Added 1 missing workflow (standardized coefficients in coefplot)
• Identified the "anchoring effect" — skills keep the agent on-task even when the content is "redundant"

Deeper Engagement: Roadmap

See ROADMAP.md for bigger-picture improvement directions:

• Gotchas-only skill variant: Test whether a stripped-down skill with just gotchas (no reference files) scores close to the full skill. If so, the references are mostly noise.
• Better test prompts: Current prompts are structured step-by-step. Real researcher interactions are messier — goal-oriented, ambiguous, multi-step. Design tests that match real use.
• Harder tests: Several tasks hit ceiling (both arms score 55/55). Need tasks that discriminate — exact numerical verification, edge cases, less-common options.
• The anchoring effect: The skill's biggest contribution isn't always syntax knowledge — it's keeping the agent focused on the correct task. This has implications for how all skills should be designed and evaluated.
• Real-world empirical test suite: Tasks based on actual published papers with replication data.
• Dynamic help file access: Let the agent query Stata's built-in help system at runtime.

Contribution Workflow Summary

1. Identify a gap: Run the eval, read judge findings from low-scoring runs, or pick a roadmap item
2. Save a baseline: --runs 5 --save on the relevant tasks
3. Make your change: Edit the reference/package doc
4. Re-run and compare: --compare against the baseline
5. Dispatch analysis agents if the results are surprising or if you want deeper insight
6. Open a PR with your results (scores, variance, before/after). See CONTRIBUTING.md for details.