| name | run |
| description | Build or load cached USearch index from QASPER, run eval pipeline, report metrics |
| user_invocable | true |
/run — Evaluate the RAG System
Build or load a cached USearch index, run evaluation against all 2,814 questions in eval_set.json, and report metrics.
Steps
rm -f ${RAG_REPORT_PATH:-/tmp/rag-eval-report.json}
PYTHONPATH=src python -m rag.pipeline
Caching
The index is cached in $RAG_INDEX_CACHE_DIR (default /tmp/rag-index-cache). Cache invalidation is output-based: chunk IDs are hashed to detect changes.
- Cache HIT (~12s): only retriever/reranker/evaluator code changed → load cached index, run queries
- Cache MISS (~80s): chunker/corpus/embeddings/config changed → full rebuild, save cache
- Force rebuild: set
RAG_FORCE_REBUILD=1
Metrics
The pipeline computes mean metrics across ALL golden labels in eval_set.json:
- NDCG@5 (primary): mean(normalized discounted cumulative gain) across all questions
- Recall@5 (secondary): mean(relevant_in_top5 / total_gold) across all questions
- MRR: mean(1/rank_of_first_hit) across all questions
- Precision@5: mean(relevant_in_top5 / 5) across all questions
Important
- This is the ONLY entry point for evaluation
- Uses USearch (HNSW, f16) for vector indexing
- Uses MLX embeddings on Apple Silicon (falls back to sentence-transformers)
- Corpus streamed from HuggingFace
allenai/qasper
