experiment-results

name	experiment-results
description	Generates a complete experiment results report from training pipeline outputs. Includes a phase-by-phase comparison table, AUC evolution analysis, technical insights, prioritized next steps, and a business-friendly section with an operational impact simulator. Trigger: When the user says "generate results", "experiment results", "create experiment report", "results report", "experiment analysis", "generate _results.md".
license	Apache-2.0
metadata	{"author":"gentleman-programming","version":"1.1"}

When to Use

• After running a series of experiments (energizados run train)
• When the user requests a results report or comparison table
• To close an experimentation cycle with analysis and recommendations

Prerequisites

• Experiment directory with outputs generated by the pipeline
• Each experiment must have reports/evaluation/evaluation_report.json
• A _experiments.md file with the phase roadmap (optional but recommended)

Critical Patterns

Input Discovery

1. Locate the experiments directory: Look in {project}/.projects/{name}/output/{version}/exp/
   • Fallback: standard framework output at output/ (runs named train-YYYYMMDD_HHMM/)
2. Locate the roadmap: Look for _experiments.md at the same level as the config
3. Read ALL evaluation_report.json files from each experiment subdirectory

Metrics to Extract (from each JSON)

metrics.auc            → AUC test
metrics.auc_val        → AUC validation
metrics.precision      → Precision @ threshold
metrics.recall         → Recall @ threshold
metrics.f1             → F1 @ threshold
metrics.threshold      → Threshold used
metrics.auc_diff       → AUC val - AUC test (overfit indicator)
metrics.confusion_matrix → TP, FP, FN, TN
metrics.cumulative_gains → Deciles and cumulative gain (for simulator)
model_info.model_class → Model type
model_info.inner_model → Inner model
calibration            → Calibration info (if present)

Generation Workflow

1. Discover experiments
   ├── ls the exp/ directory
   ├── For each subdir, read evaluation_report.json
   └── Extract key metrics

2. Group by phase (using naming convention: phaseX_expN_*)
   ├── Parse directory name
   └── Group into tables by phase

3. Identify best results
   ├── Best AUC test overall
   ├── Best AUC test per phase
   ├── Best F1
   └── Best calibrated model

4. Generate analysis
   ├── AUC evolution (ASCII chart)
   ├── What worked / what didn't / surprises
   ├── Prioritized next steps
   └── Winning pipeline

5. Generate business section (inside _results.md)
   ├── Explain metrics in non-technical language
   ├── Operational impact simulator
   └── Actionable recommendations

6. Generate business slides (separate file: _slides_negocio.md)
   ├── Translate results into presentation-ready slides
   ├── Use neutral, professional language
   └── Focus on operational impact and next steps

Output Format

Two files are generated in the same directory as _experiments.md:

1. _results.md — Full technical report.

Structure:

# Experiment Results — {PROJECT} {VERSION}
> metadata (date, guide metric, dataset)

## Executive Summary (table with best metrics)

## Full Results Table (by phase)
### PHASE 1 — {name}
### PHASE 2 — {name}
...

## AUC Test Evolution by Phase (ASCII chart)

## Key Insights
### What worked
### What did NOT work
### Surprises

## Technical Notes

## Next Steps (prioritized table)

## Winning Pipeline (YAML)

## For Business: What do these numbers mean
### What the model measures (non-technical explanation)
### Operational impact simulator
### Recommendations

## Missing Experiments

2. _slides_negocio.md — Business-facing slide deck in Markdown.

• Generated after _results.md is complete.
• Uses the same data but written as presentation slides (10–12 slides).
• Language must match the project language (Spanish if the results are in Spanish).
• See Slides Section below for detailed guidelines.

Business Section — Guidelines

The business section MUST include:

1. Metric explanations without technical jargon:

   • AUC → "out of every 100 pairs (fraud/non-fraud), the model ranks the fraud case correctly X times"
   • Precision → "out of every 100 clients the model flags as fraud, X actually committed fraud"
   • Recall → "out of every 100 real frauds, the model detects X"
   • F1 → balance between the two
   • Cumulative Gains → "inspecting the top X% of highest-risk clients, we find Y% of all frauds"

2. Operational simulator using the best model's cumulative_gains:

   • Table: "If we inspect the top 10%/20%/30%... of clients, how many frauds do we find vs. random"
   • Use deciles from evaluation_report.json
   • Estimate: inspections needed, frauds detected, false positives

3. Actionable recommendations:

   • Suggested operational threshold
   • Estimated resources required
   • False positive/negative risk

Slides Section

The _slides_negocio.md file is a presentation-ready slide deck derived from _results.md, designed for a non-technical business audience.

Generation Rules

1. Generate only after _results.md is complete — the slides are a derivative artifact.
2. Language: match the project language (e.g., Spanish for Spanish-language results).
3. Tone: neutral, professional, and presentation-oriented. Avoid:
   • Colloquialisms (e.g., "al pedo", "de onda").
   • Sarcasm or irony.
   • Overly casual phrasing (e.g., "es así de fácil").
4. Format: each slide is a level-2 heading (## Slide N: Title) separated by ---.
5. Length: 10–12 slides maximum. Keep each slide scannable (tables, bullets, short paragraphs).

Required Slides

#	Slide	Content
1	El Problema / The Problem	Business context, cost of fraud, why the model is needed.
2	Qué Hicimos / What We Did	Number of experiments, methodology, split strategy.
3	Resultado Principal / Key Results	Best AUC, Precision, Recall, F1 with plain-language interpretation.
4	Mayor Driver de Performance / Key Driver	What moved the needle most (usually feature engineering).
5	Simulador de Impacto / Impact Simulator	Concrete scenario with total clients, inspections, and frauds found vs. random.
6	Curva de Ganancia / Cumulative Gains	Decile table showing sweet spot (typically top 20–30%).
7	Recomendaciones Operativas / Operational Recommendations	Threshold, expected false-positive rate, retraining cadence, risk of false negatives.
8	Roadmap / What’s Next	Prioritized next steps with impact and effort.
9	Preguntas Frecuentes / FAQ	2–3 anticipated objections answered with data (e.g., low precision, high false positives).
10	Próximos Pasos Inmediatos / Immediate Next Steps	Pilot, integration, feedback loop.

FAQ Guidelines

Anticipate and neutrally address typical business concerns:

• "Why is precision low?" → Explain the base rate. If fraud is 9%, even a perfect model would have low precision. Show the multiplier vs. baseline.
• "Won’t most inspections be wasted?" → Contrast with random selection. Without a model, the false-positive rate is even higher because the base rate is low.
• "Is this better than what we do today?" → Provide the efficiency multiplier (e.g., 1.8x more detections per inspection).

Operational Recommendations Slide

• State the recommended threshold and its rationale (cost-benefit analysis if available).
• Frame the false-positive rate as expected and manageable, not as a failure.
• Suggest a concrete retraining frequency (e.g., monthly).
• Mention the need for complementary business rules to cover undetected frauds.

Template Reference

See assets/slides_template.md for a complete example in Spanish.

Code Examples

Extract metrics from all experiments

for f in {exp_dir}/*/reports/evaluation/evaluation_report.json; do
  dir=$(basename $(dirname $(dirname $(dirname "$f"))))
  python3 -c "
import json
d = json.load(open('$f'))
m = d['metrics']
mi = d.get('model_info', {})
cal = d.get('calibration', {})
print(f'$dir|{mi.get(\"model_class\",\"\")}|{m[\"auc\"]:.4f}|{m.get(\"auc_val\",\"NaN\"):.4f}|{m[\"precision\"]:.4f}|{m[\"recall\"]:.4f}|{m[\"f1\"]:.4f}|{m[\"threshold\"]}|{m.get(\"auc_diff\",\"NaN\"):.4f}')
"
done

Extract cumulative gains for the business simulator

python3 -c "
import json
d = json.load(open('{best_exp}/reports/evaluation/evaluation_report.json'))
gains = d['metrics']['cumulative_gains']
for pop, gain in zip(gains['cumulative_population'], gains['cumulative_gain']):
    pct = int(pop * 100)
    fraud_pct = gain * 100
    print(f'Top {pct}% inspections → {fraud_pct:.1f}% frauds detected')
"

Commands

# List available experiments
ls -1 {exp_dir}/

# Quick metrics for all experiments
for f in {exp_dir}/*/reports/evaluation/evaluation_report.json; do
  dir=$(basename $(dirname $(dirname $(dirname "$f"))))
  python3 -c "import json; d=json.load(open('$f')); m=d['metrics']; print(f'{\"$dir\"}: AUC={m[\"auc\"]:.4f} F1={m[\"f1\"]:.4f}')"
done

Resources

• Templates: See assets/ for _results.md and _slides_negocio.md template structures.
• Scripts: See scripts/ for metric extraction helpers.