원클릭으로 Manus에서 모든 스킬 실행

analyzing-experiment-results

스타7

포크2

업데이트2026년 4월 28일 10:56

Analyzing RAG experiment results with statistical rigor including pivot tables, effect sizes, and ANOVA. Use when comparing configurations, calculating significance, generating findings, or creating analysis reports. Triggers on "analyze results", "compare experiments", "statistical analysis", "generate findings".

설치

Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.

Manus에서 실행

출처

somasays

somasays/rag-experiments

GitHub 저장소 열기 Creator 저장소 보기

다운로드

Manus에서 실행

Analyzing Experiment Results

Statistical analysis of RAG experiment results.

When to Use

User says "analyze results" or "compare configurations"
Need pivot tables, effect sizes, significance tests
Generating data-driven findings

Progress Checklist

[ ] 1. Load results from summary.json files
[ ] 2. Filter out failed runs (error field present)
[ ] 3. Create pivot tables (factor × factor → metric)
[ ] 4. Calculate effect sizes (Cohen's d)
[ ] 5. Run significance tests if needed (ANOVA)
[ ] 6. Identify best/worst configurations
[ ] 7. Generate findings markdown

Core Rules

YOU MUST:

Filter out results with "error" field before analysis
Round metrics to 4 decimal places for display
Report effect sizes, not just p-values

YOU MUST NOT:

Claim significance without proper testing
Report means without variance/range

Quick Analysis

import pandas as pd
import json

def load_results(path):
    with open(path) as f:
        data = json.load(f)
    df = pd.DataFrame(data["results"])
    return df[~df.get("error", pd.Series(False)).astype(bool)]

def pivot_analysis(df, row, col, metric):
    return df.pivot_table(values=metric, index=row, columns=col, aggfunc="mean").round(4)

def best_config(df, metric="context_recall"):
    idx = df[metric].idxmax()
    return df.loc[idx]

Effect Size Interpretation

Cohen's d	Interpretation
< 0.2	Negligible
0.2 - 0.5	Small
0.5 - 0.8	Medium
> 0.8	Large

Findings Template

# Experiment [N] Findings

## Key Finding
[One sentence: "X factor had Y effect on Z metric"]

## Best Configuration
- Config: [name]
- context_recall: [value]

## Factor Analysis
| Factor | Range | Best Level |
|--------|-------|------------|
| chunking | 0.05 | sentence |
| corpus | 0.12 | clean |

## Implication
[What this means for RAG design]

References

See references/statistical_tests.md for ANOVA implementation
See references/effect_sizes.md for Cohen's d calculation
See scripts/generate_report.py for automated report generation

이 저장소의 다른 Skills

같은 저장소

building-test-corpora

somasays/rag-experiments

Building document corpora with controlled complexity levels (clean, mixed, messy) for RAG experiments. Use when creating test datasets, applying document degradation, or organizing documents for controlled experiments. Triggers on "build corpus", "create documents", "degrade documents", "test data".

2026-04-287

developing-experiment-code

somasays/rag-experiments

Developing Python experiment code with incremental validation, proper error handling, and project structure. Use when writing Python modules, creating experiment infrastructure, or setting up project scaffolding. Triggers on "create module", "write Python", "setup project", "implement feature".

2026-04-287

evaluating-with-ragas

somasays/rag-experiments

Evaluating RAG pipelines using RAGAS metrics (context_recall, context_precision, faithfulness, answer_relevancy). Use when measuring retrieval quality, comparing RAG configurations, or running LLM-as-judge evaluation. Triggers on "evaluate", "RAGAS", "measure recall", "context precision".

2026-04-287

running-rag-experiments

somasays/rag-experiments

Running RAG experiment configurations with checkpointing and error recovery. Use when executing batch experiments, handling experiment failures gracefully, or resuming interrupted experiment runs. Triggers on "run experiment", "execute configurations", "resume from checkpoint".

2026-04-287

name	analyzing-experiment-results
description	Analyzing RAG experiment results with statistical rigor including pivot tables, effect sizes, and ANOVA. Use when comparing configurations, calculating significance, generating findings, or creating analysis reports. Triggers on "analyze results", "compare experiments", "statistical analysis", "generate findings".

Analyzing Experiment Results

Statistical analysis of RAG experiment results.

When to Use

User says "analyze results" or "compare configurations"
Need pivot tables, effect sizes, significance tests
Generating data-driven findings

Progress Checklist

[ ] 1. Load results from summary.json files
[ ] 2. Filter out failed runs (error field present)
[ ] 3. Create pivot tables (factor × factor → metric)
[ ] 4. Calculate effect sizes (Cohen's d)
[ ] 5. Run significance tests if needed (ANOVA)
[ ] 6. Identify best/worst configurations
[ ] 7. Generate findings markdown

Core Rules

YOU MUST:

Filter out results with "error" field before analysis
Round metrics to 4 decimal places for display
Report effect sizes, not just p-values

YOU MUST NOT:

Claim significance without proper testing
Report means without variance/range

Quick Analysis

import pandas as pd
import json

def load_results(path):
    with open(path) as f:
        data = json.load(f)
    df = pd.DataFrame(data["results"])
    return df[~df.get("error", pd.Series(False)).astype(bool)]

def pivot_analysis(df, row, col, metric):
    return df.pivot_table(values=metric, index=row, columns=col, aggfunc="mean").round(4)

def best_config(df, metric="context_recall"):
    idx = df[metric].idxmax()
    return df.loc[idx]

Effect Size Interpretation

Cohen's d	Interpretation
< 0.2	Negligible
0.2 - 0.5	Small
0.5 - 0.8	Medium
> 0.8	Large

Findings Template

# Experiment [N] Findings

## Key Finding
[One sentence: "X factor had Y effect on Z metric"]

## Best Configuration
- Config: [name]
- context_recall: [value]

## Factor Analysis
| Factor | Range | Best Level |
|--------|-------|------------|
| chunking | 0.05 | sentence |
| corpus | 0.12 | clean |

## Implication
[What this means for RAG design]

References

See references/statistical_tests.md for ANOVA implementation
See references/effect_sizes.md for Cohen's d calculation
See scripts/generate_report.py for automated report generation