| name | experiment-analysis |
| description | Analyze mvp-engine experiment outputs and produce a structured Markdown report grounded in run artifacts, logs, checkpoints, optional eval files, and baseline comparisons. |
Experiment Analysis
Goal
Produce one evidence-grounded Markdown report that helps decide what to do next:
- summarize run setup, training dynamics, checkpoints, and failures;
- compare runs only when metrics are comparable;
- use repo-native artifacts as primary evidence;
- use external eval files as optional additional evidence;
- clearly separate direct evidence from inference.
Required Inputs
Identify these before writing:
- one or more
outputs/<run_id>/ directories;
config_<run_id>.yaml, log_<run_id>.log, and checkpoints/ when present;- optional
results*.json, metrics*.json, samples*.jsonl,
predictions*.jsonl, or predictions*.csv;
- optional baseline or reference runs;
- target report path when the user wants an existing report updated.
Ask the user only when the run path, comparison target, or report destination is
ambiguous.
Workflow
1. Discover Artifacts
Start from repo-native outputs:
rg --files outputs -g 'config_*.yaml' -g 'log_*.log'
find outputs -path '*/checkpoints/*' -maxdepth 3 -type d
Read references/mvp-engine-artifacts.md when artifact layout or discovery is
unclear.
2. Extract Deterministic Run Summaries
Run the bundled helper for each run directory:
python3 skills/experiment/experiment-analysis/scripts/summarize_run.py \
--run-dir outputs/<run_id>
Use the helper output as notes. Read raw logs directly when warnings, errors,
metric anomalies, or message context matter.
3. Analyze Evidence
For each run, collect:
- config metadata: recipe, engine, model, loop policy, total steps, git info;
- latest and best logged metrics;
- warnings, errors, stalls, regressions, and throughput anomalies;
- checkpoint inventory and latest logged step versus latest checkpoint;
- optional external evaluation metrics and sample-level failure examples.
For multi-run comparisons, normalize metric names and compare only values with
the same definition and comparable step selection.
4. Write One Canonical Report
Read references/report-template.md before drafting the final report.
The report should include:
- analysis scope and decision questions;
- exact artifact paths;
- headline conclusions;
- run metadata and training dynamics;
- checkpoint audit;
- evaluation and comparison tables when evidence exists;
- representative failures when sample-level files exist;
- recommendations and missing evidence.
5. Use Multi-Agent Mode Only When Helpful
If the task includes multiple independent workstreams, read
references/multi-agent-plan.md. Use subagents only when explicitly available
and useful for parallel artifact discovery, metric extraction, comparison, or
error analysis.
Output
- State the report path if a file was written.
- State which run directories and external artifacts were analyzed.
- Summarize the strongest conclusions and biggest evidence gaps.
- State which extraction commands ran and whether any artifacts were missing.
Read On Demand
references/mvp-engine-artifacts.md: artifact layout and discovery commands.references/report-template.md: final Markdown report structure.references/multi-agent-plan.md: optional parallel analysis split.
