| name | rag-ops |
| description | Deep RAG application-layer operational intuition — chunking (late chunking, contextual retrieval), embedding-model landscape, reranker economics (ColBERT, ColPali), hybrid retrieval (BM25 + dense + RRF, SPLADE-v3), evaluation (RAGAS, judge bias), grounding, long-context-vs-RAG. Load when designing chunking, embedding-model selection, reranker integration, eval design, hallucination audits, or long-context-vs-RAG sizing. Skip for ANN index choice (use `vector-search`), embedding fine-tuning, or prompt engineering. Pairs with `vector-search`. Triggers on: "late chunking", "contextual retrieval", "MTEB", "BGE-M3", "Matryoshka embeddings", "SPLADE-v3", "RRF", "ColBERTv2", "ColPali", "HyDE", "RAGAS", "groundedness", "lost in the middle", "Self-RAG", "CRAG".
|
RAG Operational Guide
Concise operational pointers for deep RAG application-layer tuning and incident diagnosis.
Assumes you already know what RAG is, basic embedding usage, and that you have a vector store. This skill covers the operational layer — the parts models tend to gloss over: chunking economics, the 2026 embedding-model landscape, reranker tradeoffs, hybrid fusion math, evaluation methodology and judge bias, hallucination/grounding, drift handling, long-context-vs-RAG decisions — current as of late 2025/early 2026.
When to use
Load when the question is about:
- Choosing chunk size / splitter (fixed / recursive / semantic / token-aware / late) for a known embed model
- Tuning chunking against
embed_model.max_seq_length (e.g., 512 vs 8192)
- Contextual retrieval (Anthropic chunk augmentation) vs late chunking (Jina) vs vanilla
- Picking 2026 embedding model from MTEB top set under cost/latency/dim constraints
- Matryoshka truncation (e.g., 3072 → 256 dim) for ANN cost reduction
- Hybrid (dense + BM25 / SPLADE) with RRF and avoiding score-normalization mistakes
- Adding a reranker stage (Cohere v3.5, BGE v2-m3, Jina v3) and budgeting 100–600 ms
- HyDE / multi-query / decomposition vs single-shot when latency-sensitive
- Building a RAGAS / TruLens eval harness with golden set, NDCG@k / MRR / Recall@k
- Hallucination detection via faithfulness, NLI, citation grounding, Anthropic Citations API
- Re-embed planning when changing model or pipeline; hot/cold index split
- Long-context (Claude / Gemini 1M) vs RAG; mitigating "lost in the middle" / "context rot"
- Multimodal / visual document retrieval with ColPali, ColQwen2, ColSmol
Do NOT load for: ANN index data-structure choice (use vector-search), vector-DB shard layout, LLM generation/prompt engineering not bound to retrieved context, agent / tool-use orchestration without retrieval semantics, fine-tuning embedding models from scratch.
Chunking — concrete defaults
- Fixed-size: deprecated default; baseline only. Rule of thumb: ~50% of
embed_model.max_seq_length with 10–20% overlap.
- Recursive character split (LangChain
RecursiveCharacterTextSplitter): hierarchical separators ["\n\n","\n"," ",""]. Default chunk_size=512, chunk_overlap=64 is a starting point — re-tune against the model's tokenizer.
- Token-aware: count with the model's tokenizer (
tiktoken for OpenAI, AutoTokenizer for HF, voyageai.Client.count_tokens for Voyage). 1 token ≠ 1 word for non-Latin; chunking on chars overruns max_seq_length for CJK/code.
- Sentence / semantic chunking: split on sentence boundaries, then merge until a similarity drop ("breakpoint percentile" ~95) — LlamaIndex
SemanticSplitterNodeParser. Higher recall on argumentative text, slower index.
- Late chunking (Jina, arXiv 2409.04701): embed the entire long doc through the transformer first, then mean-pool over token spans → chunk vectors that retain document context. Average +3.6% relative retrieval over naive sentence-boundary chunking; native in
jina-embeddings-v3 (8192 ctx). No retraining needed.
- Anthropic Contextual Retrieval: prepend a 50–100 token Claude-Haiku-generated chunk-context blurb before embedding and BM25 indexing. Prompt-cached cost ≈ $1.02 per 1M document tokens. Reported failure-rate reductions (top-20): 35% with contextual embeddings alone (5.7% → 3.7%), 49% combined with contextual BM25 (→ 2.9%), 67% adding a reranker (→ 1.9%).
- Optimal chunk size is bounded by the model's effective context: BGE-M3 / Jina v3 (8192), E5-Mistral (32k but degrades), text-embedding-3-large (8191). Set chunks ≤ 50% of effective ctx so query + chunk fits at rerank time.
Embedding model selection — durable axes
The current MTEB leaderboard rankings rotate every few months — re-check before committing. Choose along durable axes, not the leaderboard's top row:
- Context length vs your post-chunking length budget (512 / 8192 / 32k typical).
- Dense-only vs hybrid output (e.g., BGE-M3 emits dense + sparse + multi-vector in one forward pass — single-model hybrid).
- Matryoshka support — store the full-dim vector, query at any nested dim (256 / 512 / 1024). Enables tiered search: short-vector ANN candidate gen → full-vector rerank. Renormalize after truncation if cosine.
- Asymmetric query/doc vs symmetric — shared vector space matters for retrieval-as-search vs symmetric-similarity tasks.
- Multilingual / multimodal coverage if non-English or vision-document-heavy.
- Self-host vs hosted — cost model flips with QPS; hosted dominates at low QPS, self-host at high QPS once a GPU is paid for.
Reranking economics
- Bi-encoder (your retriever): ~5–20 ms/query, no per-doc inference at query time.
- Cross-encoder reranker: scores
[query, doc] jointly. Latency scales with top_k_to_rerank × seq_len. Typical 2-stage: retrieve top 50–100, rerank to top 10.
Reranker model choice (BGE / Cohere / Jina / Voyage families) and their per-query pricing turn over every few months — re-check current model cards. Budget on the latency math (per-query ms × QPS × replicas) rather than headline nDCG deltas.
ColBERTv2 / Jina-ColBERT-v2 (late interaction): MaxSim per query token over doc tokens. PLAID-style residual-compression cuts storage 6–10×. Sweet spot when you need reranker-quality at retriever-time latency on huge corpora.
Hybrid retrieval
Standard recipe: dense top-k + BM25 top-k → RRF fuse. RRF(d) = Σ 1/(k + rank_i(d)) with k=60 (empirically robust per Cormack 2009). RRF is rank-based, so it does NOT need score normalization — common pitfall is min-max-normalizing dense cosine and BM25 raw scores then weighted-summing; BM25 is unbounded and dataset-dependent, breaking the average.
Learned sparse: SPLADE-v3 (lexical-expansion via MLM head, term weights) bridges dense recall and BM25 lexical precision. BGE-M3 produces dense + sparse + multi-vector in one forward pass — single-model hybrid.
Typical fused weighting: 0.6 dense / 0.4 sparse. Fuse top 50 from each; rerank top 30 fused → top 10.
Query transformations
- HyDE: LLM generates a hypothetical answer; embed and retrieve against it. Wins on vocabulary mismatch (medical, legal). Adds 1 LLM call (~300–800 ms small model). Hurts on factual lookup where the generated answer fabricates wrong terminology.
- Multi-query / RAG-Fusion: generate N rewrites, retrieve each, RRF-fuse. Wins on ambiguous queries; cost is N× retriever calls + 1 LLM call.
- Decomposition: break complex query into sub-queries, retrieve per sub, aggregate. Mandatory for multi-hop QA; latency cost compounds.
Production pattern: route via a small classifier — short factual → direct; vocabulary-gap → HyDE; multi-hop → decompose. ≤1B router (or logistic regression on query features) keeps overhead < 50 ms.
Evaluation
- Retrieval-only: NDCG@k (graded relevance, position-discounted), MRR (first-hit reciprocal), Recall@k (coverage), Hit@k (binary). Report at k = 5, 10, 20.
- RAGAS metrics:
- Faithfulness = claims-extracted-from-answer / claims-supported-by-context (NLI-prompted).
- Answer relevancy = similarity of answer to LLM-rephrased queries derived from the answer.
- Context precision (rank-aware: are relevant chunks at top?).
- Context recall (needs ground truth).
- TruLens RAG triad: context relevance, groundedness, answer relevance — all LLM-as-judge.
- LLM-judge biases: length (verbose answers under-rated
10% by some judges, over-rated by others), positional, self-preference, format. Mitigations: ensemble of 3 judges (+8% accuracy), pairwise prompts with order-swap, calibrated rubrics with anchor examples, capping max_tokens to neutralize length.
- Golden set: 200–500 (query, ideal-answer, supporting-doc-ids) tuples drawn from production logs (sampled, redacted, human-validated). Refresh quarterly.
Hallucination / attribution
- Faithfulness via NLI: split answer into atomic claims (LLM), each claim → (entail / neutral / contradict) over retrieved context.
groundedness < 1 → hallucination.
- Citation grounding via Anthropic Citations API (Messages API,
citations: {enabled: true}): Claude returns sentence/passage spans into provided documents. Reported case (Endex): source-hallucination 10% → 0%, +20% citations per response. Caveat: citations don't fix factual errors when the source itself is wrong.
- Patronus Lynx, RAGTruth, MiniCheck — open-source hallucination detectors; combine with judge ensemble for cheap online QA.
Index freshness
- Embedding-drift sources: tokenizer change, model version bump, normalization toggle, preprocessing edit, re-embedding only a subset. Rule: one index = exactly one (
model_version, pipeline_hash, normalization) tuple. Pin via pipeline_id field on every vector.
- Hot/cold split: hot index — recent N days, frequent incremental upserts; cold index — bulk-rebuilt off-peak (3 AM weekends), promoted atomically. Pinecone, Weaviate, Qdrant support zero-downtime alias swap.
- Re-embed cost: per-token embedding API pricing changes — multiply current rate by token count when budgeting a model swap. OSS local re-embed cost is dominated by GPU-hours, not per-token rates.
- Incremental indexing: emit
{doc_id, content_hash, last_modified} events; only re-embed when content_hash changes. Tombstone deletes; never partially update a chunked doc — re-chunk the whole document.
Long context vs RAG
- Long-context wins: small corpora (≤ 200 pages), unique single-document tasks, multi-hop reasoning where chunking would sever dependencies, exploratory analysis.
- RAG wins: large corpora (>1M tokens), low-latency / high-QPS, cost-sensitive, requires citation per answer, frequently updated content.
- "Lost in the middle" / context rot: accuracy degrades for content placed mid-context. Frontier long-context models do well on synthetic NIAH but lose meaningfully on realistic multi-fact recall. Re-check current numbers against your task — published headline scores rarely match production retrieval quality.
- Cost / latency: a full 1M-token request is orders-of-magnitude slower and more expensive per query than RAG at equivalent retrieval quality. Hybrid routing (cheap-RAG-first, escalate-to-long-context) is the dominant pattern.
Multimodal RAG
- ColPali (ICLR 2025): VLM (PaliGemma backbone) → multi-vector embeddings of page-image patches. Late-interaction MaxSim between query token vectors and patch vectors. Eliminates OCR / layout-parse / text-chunking pipelines for PDFs, slides, screenshots.
- ColQwen2 — Qwen2-VL backbone, same recipe; ColSmol — small variant for edge.
- Storage cost is high (multi-vector per page); use binary / scalar quantization, PLAID two-stage filter.
- Use when documents are layout-heavy (financial filings, scientific PDFs, scanned forms).
Failure modes
- Needle-in-haystack: evaluate with NoLiMa / RULER, not synthetic NIAH (saturated).
- Similarity-vs-relevance gap: cosine high, intent wrong (e.g., antonyms cluster nearby in word2vec-derived spaces). Mitigate with reranker + query intent classification.
- Vocabulary mismatch: query uses common terms, docs use jargon (or vice-versa). Mitigate with HyDE or hybrid BM25/SPLADE.
- Multi-hop: single retrieval insufficient; needs decomposition + iterative retrieval (Self-RAG
retrieve token, CRAG retrieval-quality evaluator with web fallback).
- Negation / temporal: embeddings ignore "not" and tense; route to a structured filter when detected.
Production patterns
- Caching layers: query-embedding cache (key =
hash(query) + model_version), retriever-result cache (TTL ≤ index-rebuild interval), reranker cache (key = hash(query, top_k_doc_ids)), answer cache only if generation is deterministic.
- Retrieval-log mining: sample 1% of queries with retrieved docs and downstream signals (click, thumbs-up, follow-up rephrasing) → weekly golden-set augmentation.
- Drift monitoring: track P50 retrieval-score distribution per day; alert on >2σ shift. Track answer-length distribution and faithfulness on a held-out probe set.
- Self-RAG / CRAG in prod: lightweight retrieval-quality classifier; on "ambiguous" → fall back to web / broader index; on "irrelevant" → skip retrieval.
Authoritative references
Guardrails
Before recommending a non-trivial RAG-pipeline change (chunking strategy, embed-model swap, reranker addition, hybrid weights, re-embed):
- Quote the parameter / approach and any default
- Cite the originating paper / blog / docs
- Make the recommendation conditional on observed evidence (NDCG@k, MRR, faithfulness on a golden set, latency P99) — never blanket-tune
- Verify model versions and pipeline IDs — many features are release-gated and rotate fast.
Tuning without measurement is worse than defaults.
