rag-architect

name	rag-architect
description	Design and audit Retrieval-Augmented Generation (RAG) pipelines — chunking strategy, embedding choice, hybrid search, retrieval evaluation, and failure diagnosis. Use when asked to "build a RAG system", "fix retrieval quality", "improve RAG accuracy", "chunk documents", "add semantic search", or when an LLM keeps hallucinating despite having access to docs. Do NOT use for: fine-tuning decisions — that is a separate trade-off.
origin	yamtam-original
license	MIT © 2025 Vũ Văn Tâm
version	1.0.0
compatibility	Any LLM stack. Patterns apply to LangChain, LlamaIndex, custom pipelines.

When to Use

• Use when: building document Q&A, knowledge base search, or context injection
• Use when: LLM answers are hallucinated despite docs being "in the system"
• Use when: retrieval returns irrelevant chunks
• Use when: auditing an existing RAG pipeline before production
• Do NOT use for: prompt template design — use prompt-engineering skill
• Do NOT use for: LLM UI (streaming, states) — use llm-ui-patterns skill

---

RAG Failure Modes

73% of RAG failures trace to retrieval; 80% of retrieval failures trace to chunking.

Failure	Symptom	Root cause
Context miss	LLM says "I don't know" but doc exists	Chunk too large — answer buried
Context flood	Correct answer lost in noise	Retrieving too many chunks
Semantic split	Answer split across chunk boundary	Fixed-size chunking ignores structure
Stale retrieval	Outdated answer served	No timestamp filter in query
Cross-doc confusion	Answer blends facts from multiple docs	Missing source-isolation metadata

---

Chunking Strategy

Default: 512 tokens + 10–20% overlap

chunk_size    = 512 tokens  (≈ 1-2 paragraphs)
chunk_overlap = 10–20%      (51–102 tokens)

• Overlap prevents answers split at chunk boundaries
• 512 is the empirical sweet spot: large enough for context, small enough for precision

Choose by content type

Content type	Strategy	Size
Narrative prose	Fixed-size + overlap	512 tokens, 15% overlap
Technical docs	Sentence-level	2–5 sentences per chunk
Code	Function-level	One function per chunk
FAQ / Q&A pairs	Semantic unit	One Q+A pair per chunk
Legal / contracts	Section-level	One clause per chunk
Tables / structured data	Row or block	One logical row/block

Metadata-aware chunking (+15–25% retrieval accuracy)

Every chunk must carry:

{
  "source": "handbook.pdf",
  "section": "Chapter 3 — Onboarding",
  "page": 14,
  "created_at": "2025-03-01",
  "doc_type": "policy",
  "chunk_index": 7,
  "total_chunks": 42
}

Use metadata for pre-filtering before vector search — eliminates irrelevant domains.

---

Retrieval Architecture

Hybrid search (dense + sparse) — recommended default

Query
  │
  ├── Dense search (embedding similarity)   → top-K candidates
  ├── Sparse search (BM25 / keyword)        → top-K candidates
  │
  └── Reciprocal Rank Fusion (RRF)          → merged, re-ranked result set

Dense search: captures semantic meaning ("employee time off" matches "vacation policy") Sparse search: captures exact terminology ("RFC 2119", "GDPR Art. 17") — dense alone misses these

RRF fusion formula: score = Σ 1 / (rank_i + k), k=60 is standard default.

Re-ranking (optional second pass)

After RRF, run a cross-encoder re-ranker on top-20 → return top-5. Cost: 1 extra inference call. Gain: removes false positives missed by embedding similarity.

Top-K selection

Use case	top-K
Precise Q&A	3–5
Summarization	8–15
Research assistant	15–25

Higher K = more context = higher cost + risk of context flood.

---

Retrieval Evaluation — From Day One

Never skip eval. Measure at pipeline build time, not after ship.

Metrics to track

Metric	What it measures	Target
Hit Rate	Answer chunk in top-K	> 0.85
MRR (Mean Reciprocal Rank)	How high the answer chunk ranks	> 0.70
Context Relevance	Retrieved chunks relevant to query	> 0.80
Answer Faithfulness	LLM answer grounded in chunks	> 0.90
Answer Relevance	Answer addresses the question	> 0.85

Tools: Ragas (open source), LlamaIndex Eval, ARES.

Minimum eval dataset

• 50 questions minimum before any production claim
• Include: factual, multi-hop, negative (answer not in docs), edge-case

---

Pipeline Checklist

□ Chunking strategy chosen for content type (not default everywhere)
□ Chunk metadata includes: source, section, timestamp, doc_type
□ Hybrid search (dense + sparse) implemented — or documented why not
□ top-K tuned per use case (not fixed at 10 for everything)
□ Retrieval eval dataset built (≥ 50 questions)
□ Hit Rate ≥ 0.85 measured before ship claim
□ Answer Faithfulness ≥ 0.90 measured
□ Stale document handling: timestamp filter or re-indexing schedule defined

---

Anti-Fake-Pass Rules

Before claiming "RAG is working", you MUST show:

• Hit Rate and MRR measured on an eval dataset (not spot-checked)
• Chunking strategy justified for the actual content type
• Metadata schema defined (source, section, timestamp at minimum)
• Hybrid search in place OR documented why dense-only suffices
• top-K value justified, not left at arbitrary default

Reference: gates/anti-fake-pass-gate.md