running-rag-experiments

name	running-rag-experiments
description	Running RAG experiment configurations with checkpointing and error recovery. Use when executing batch experiments, handling experiment failures gracefully, or resuming interrupted experiment runs. Triggers on "run experiment", "execute configurations", "resume from checkpoint".

Running RAG Experiments

Execute experiment configurations with proper isolation, checkpointing, and error recovery.

When to Use

• User says "run experiment" or "execute configurations"
• Running multiple RAG pipeline configurations
• Need to resume an interrupted experiment run
• Collecting metrics across configurations

Progress Checklist

Copy this checklist and mark items as you complete them:

[ ] 1. Load checkpoint (if resuming)
[ ] 2. Verify dependencies (Ollama running, API keys set)
[ ] 3. For each configuration:
    [ ] a. Skip if already in checkpoint
    [ ] b. Run experiment
    [ ] c. Log result or error
    [ ] d. Save checkpoint immediately
    [ ] e. Cleanup resources
[ ] 4. Save final summary.json
[ ] 5. Report success/failure counts

Core Rules

YOU MUST:

• Save checkpoint after EVERY experiment run, not just at the end
• Continue to next configuration on error, never crash entirely
• Call gc.collect() between experiments to prevent memory leaks
• Filter out already-completed configurations when resuming

YOU MUST NOT:

• Skip checkpoint saves to "optimize" performance
• Let a single failure crash the entire batch
• Assume previous results exist without checking

Workflow

Load checkpoint → Filter completed → Run remaining → Save checkpoint each → Final summary

Validation

Run scripts/validate_experiment_setup.py before starting experiments:

python scripts/validate_experiment_setup.py

Expected output: "All checks passed. Ready to run experiments."

References

• See references/experiment_runner.py for complete implementation
• See references/checkpoint_format.md for checkpoint JSON structure
• See scripts/validate_experiment_setup.py for pre-flight checks