analyzing-experiment-results

name	analyzing-experiment-results
description	Analyzing RAG experiment results with statistical rigor including pivot tables, effect sizes, and ANOVA. Use when comparing configurations, calculating significance, generating findings, or creating analysis reports. Triggers on "analyze results", "compare experiments", "statistical analysis", "generate findings".

Analyzing Experiment Results

Statistical analysis of RAG experiment results.

When to Use

• User says "analyze results" or "compare configurations"
• Need pivot tables, effect sizes, significance tests
• Generating data-driven findings

Progress Checklist

[ ] 1. Load results from summary.json files
[ ] 2. Filter out failed runs (error field present)
[ ] 3. Create pivot tables (factor × factor → metric)
[ ] 4. Calculate effect sizes (Cohen's d)
[ ] 5. Run significance tests if needed (ANOVA)
[ ] 6. Identify best/worst configurations
[ ] 7. Generate findings markdown

Core Rules

YOU MUST:

• Filter out results with "error" field before analysis
• Round metrics to 4 decimal places for display
• Report effect sizes, not just p-values

YOU MUST NOT:

• Claim significance without proper testing
• Report means without variance/range

Quick Analysis

import pandas as pd
import json

def load_results(path):
    with open(path) as f:
        data = json.load(f)
    df = pd.DataFrame(data["results"])
    return df[~df.get("error", pd.Series(False)).astype(bool)]

def pivot_analysis(df, row, col, metric):
    return df.pivot_table(values=metric, index=row, columns=col, aggfunc="mean").round(4)

def best_config(df, metric="context_recall"):
    idx = df[metric].idxmax()
    return df.loc[idx]

Effect Size Interpretation

Cohen's d	Interpretation
< 0.2	Negligible
0.2 - 0.5	Small
0.5 - 0.8	Medium
> 0.8	Large

Findings Template

# Experiment [N] Findings

## Key Finding
[One sentence: "X factor had Y effect on Z metric"]

## Best Configuration
- Config: [name]
- context_recall: [value]

## Factor Analysis
| Factor | Range | Best Level |
|--------|-------|------------|
| chunking | 0.05 | sentence |
| corpus | 0.12 | clean |

## Implication
[What this means for RAG design]

References

• See references/statistical_tests.md for ANOVA implementation
• See references/effect_sizes.md for Cohen's d calculation
• See scripts/generate_report.py for automated report generation