메뉴
Evaluate RAG quality: run Precision@10 and MRR against a bundle, interpret the scores, and compare CI-built vs locally-built bundles. Use when assessing bundle quality, comparing backends, or investigating a regressed search result.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Patterns for keeping READMEs accurate as code evolves. Focused on structural consistency, not prose style. Use when editing a README with ASCII tree diagrams, version callouts, config examples, or cross-document line ranges. Avoids generic "write clearly" advice — targets the concrete rot patterns that accumulate in docs-heavy repos.
Aggressive optimization patterns for atlas. Overrides standard Python defaults with zero-dep, startup-time, and memory-efficient alternatives. Honed for a project that runs a read-only parquet-backed RAG bundle loaded into numpy at startup and queried via MCP. Use when making any code change — these patterns take precedence over generic Python best practices.
Write and maintain Architecture Decision Records (ADRs) for the atlas project. Covers ADR format, where to store them, when to create one, and examples based on past decisions. Use when making a significant technical decision that should be recorded for future maintainers.
Add a new MCP tool to the atlas-fs or atlas-rag server. Covers tool definition, handler function, parameter schema, and register call. Use when implementing a new feature that needs to be exposed as an MCP tool.
Build the portable RAG bundle: clone docs, chunk, embed, package tarball. Use when the task involves building a bundle locally (`atlas-build`), smoke testing a bundle (`atlas-smoke`), or adding a new publication.
Manage CI bundle releases: manual dispatch, tag naming, monitoring the build-bundle workflow, promoting a release, and rolling back. Use when releasing a new bundle, debugging a CI failure, or rolling back to a previous bundle.
| name | atlas-rag-evaluation |
| description | Evaluate RAG quality: run Precision@10 and MRR against a bundle, interpret the scores, and compare CI-built vs locally-built bundles. Use when assessing bundle quality, comparing backends, or investigating a regressed search result. |
List available bundles:
ls data/bundles/
Pick the one you want to evaluate.
# Point BUNDLE_DIR at the bundle and run
BUNDLE_DIR=data/bundles/<timestamp> uv run atlas-eval
If the eval script is already registered as a console script. Otherwise run directly:
BUNDLE_DIR=data/bundles/<timestamp> uv run python atlas/evaluate_rag.py
The output looks like:
Query: "how to create an incident"
P@10: 0.90 MRR: 1.0000
...
Mean P@10 over 10 queries: 0.89
Mean MRR over 10 queries: 0.94
Precision@10 (P@10): Fraction of the top-10 results that are relevant. A score of 0.89 means ~9 of 10 returned chunks are on-topic.
Mean Reciprocal Rank (MRR): How high the first relevant result appears. 1.0 means the top result is always relevant; 0.94 means the first relevant result is typically in position 1–2.
| Metric | Minimum | Target | Excellent |
|---|---|---|---|
| Precision@10 | 0.75 | 0.85 | 0.95+ |
| MRR | 0.80 | 0.90 | 0.97+ |
To compare, build two bundles with the same model on CI and locally:
# Local (MLX)
uv run atlas-build --model Xenova/bge-small-en-v1.5
BUNDLE_DIR=data/bundles/<local> uv run atlas-eval
# Download CI bundle from GitHub Releases
gh release download <tag> --pattern '*.tar.zst'
tar --zstd -xf atlas-bundle-*.tar.zst -C /tmp/ci-bundle
BUNDLE_DIR=/tmp/ci-bundle uv run atlas-eval
Compare the mean scores. Small differences (<0.02) are expected due to float rounding.
| Delta | Meaning |
|---|---|
| <0.01 | Negligible — bundles are equivalent |
| 0.01–0.03 | Minor — likely from int8 quantisation noise |
| 0.03–0.05 | Moderate — investigate chunk boundary differences |
| >0.05 | Significant — model or data divergence |
README.md §11 — Validate RAG qualityatlas/evaluate_rag.py — The evaluation script