rag-ops

name

rag-ops

description

Deep RAG application-layer operational intuition — chunking (late chunking, contextual retrieval), embedding-model landscape, reranker economics (ColBERT, ColPali), hybrid retrieval (BM25 + dense + RRF, SPLADE-v3), evaluation (RAGAS, judge bias), grounding, long-context-vs-RAG. Load when designing chunking, embedding-model selection, reranker integration, eval design, hallucination audits, or long-context-vs-RAG sizing. Skip for ANN index choice (use `vector-search`), embedding fine-tuning, or prompt engineering. Pairs with `vector-search`. Triggers on: "late chunking", "contextual retrieval", "MTEB", "BGE-M3", "Matryoshka embeddings", "SPLADE-v3", "RRF", "ColBERTv2", "ColPali", "HyDE", "RAGAS", "groundedness", "lost in the middle", "Self-RAG", "CRAG".

RAG Operational Guide

Concise operational pointers for deep RAG application-layer tuning and incident diagnosis.

Assumes you already know what RAG is, basic embedding usage, and that you have a vector store. This skill covers the operational layer — the parts models tend to gloss over: chunking economics, the 2026 embedding-model landscape, reranker tradeoffs, hybrid fusion math, evaluation methodology and judge bias, hallucination/grounding, drift handling, long-context-vs-RAG decisions — current as of late 2025/early 2026.

When to use

Load when the question is about:

• Choosing chunk size / splitter (fixed / recursive / semantic / token-aware / late) for a known embed model
• Tuning chunking against embed_model.max_seq_length (e.g., 512 vs 8192)
• Contextual retrieval (Anthropic chunk augmentation) vs late chunking (Jina) vs vanilla
• Picking 2026 embedding model from MTEB top set under cost/latency/dim constraints
• Matryoshka truncation (e.g., 3072 → 256 dim) for ANN cost reduction
• Hybrid (dense + BM25 / SPLADE) with RRF and avoiding score-normalization mistakes
• Adding a reranker stage (Cohere v3.5, BGE v2-m3, Jina v3) and budgeting 100–600 ms
• HyDE / multi-query / decomposition vs single-shot when latency-sensitive
• Building a RAGAS / TruLens eval harness with golden set, NDCG@k / MRR / Recall@k
• Hallucination detection via faithfulness, NLI, citation grounding, Anthropic Citations API
• Re-embed planning when changing model or pipeline; hot/cold index split
• Long-context (Claude / Gemini 1M) vs RAG; mitigating "lost in the middle" / "context rot"
• Multimodal / visual document retrieval with ColPali, ColQwen2, ColSmol

Do NOT load for: ANN index data-structure choice (use vector-search), vector-DB shard layout, LLM generation/prompt engineering not bound to retrieved context, agent / tool-use orchestration without retrieval semantics, fine-tuning embedding models from scratch.

Chunking — concrete defaults

• Fixed-size: deprecated default; baseline only. Rule of thumb: ~50% of embed_model.max_seq_length with 10–20% overlap.
• Recursive character split (LangChain RecursiveCharacterTextSplitter): hierarchical separators ["\n\n","\n"," ",""]. Default chunk_size=512, chunk_overlap=64 is a starting point — re-tune against the model's tokenizer.
• Token-aware: count with the model's tokenizer (tiktoken for OpenAI, AutoTokenizer for HF, voyageai.Client.count_tokens for Voyage). 1 token ≠ 1 word for non-Latin; chunking on chars overruns max_seq_length for CJK/code.
• Sentence / semantic chunking: split on sentence boundaries, then merge until a similarity drop ("breakpoint percentile" ~95) — LlamaIndex SemanticSplitterNodeParser. Higher recall on argumentative text, slower index.
• Late chunking (Jina, arXiv 2409.04701): embed the entire long doc through the transformer first, then mean-pool over token spans → chunk vectors that retain document context. Average +3.6% relative retrieval over naive sentence-boundary chunking; native in jina-embeddings-v3 (8192 ctx). No retraining needed.
• Anthropic Contextual Retrieval: prepend a 50–100 token Claude-Haiku-generated chunk-context blurb before embedding and BM25 indexing. Prompt-cached cost ≈ $1.02 per 1M document tokens. Reported failure-rate reductions (top-20): 35% with contextual embeddings alone (5.7% → 3.7%), 49% combined with contextual BM25 (→ 2.9%), 67% adding a reranker (→ 1.9%).
• Optimal chunk size is bounded by the model's effective context: BGE-M3 / Jina v3 (8192), E5-Mistral (32k but degrades), text-embedding-3-large (8191). Set chunks ≤ 50% of effective ctx so query + chunk fits at rerank time.

Embedding model selection — durable axes

The current MTEB leaderboard rankings rotate every few months — re-check before committing. Choose along durable axes, not the leaderboard's top row:

• Context length vs your post-chunking length budget (512 / 8192 / 32k typical).
• Dense-only vs hybrid output (e.g., BGE-M3 emits dense + sparse + multi-vector in one forward pass — single-model hybrid).
• Matryoshka support — store the full-dim vector, query at any nested dim (256 / 512 / 1024). Enables tiered search: short-vector ANN candidate gen → full-vector rerank. Renormalize after truncation if cosine.
• Asymmetric query/doc vs symmetric — shared vector space matters for retrieval-as-search vs symmetric-similarity tasks.
• Multilingual / multimodal coverage if non-English or vision-document-heavy.
• Self-host vs hosted — cost model flips with QPS; hosted dominates at low QPS, self-host at high QPS once a GPU is paid for.

Reranking economics

• Bi-encoder (your retriever): ~5–20 ms/query, no per-doc inference at query time.
• Cross-encoder reranker: scores [query, doc] jointly. Latency scales with top_k_to_rerank × seq_len. Typical 2-stage: retrieve top 50–100, rerank to top 10.

Reranker model choice (BGE / Cohere / Jina / Voyage families) and their per-query pricing turn over every few months — re-check current model cards. Budget on the latency math (per-query ms × QPS × replicas) rather than headline nDCG deltas.

ColBERTv2 / Jina-ColBERT-v2 (late interaction): MaxSim per query token over doc tokens. PLAID-style residual-compression cuts storage 6–10×. Sweet spot when you need reranker-quality at retriever-time latency on huge corpora.

Hybrid retrieval

Standard recipe: dense top-k + BM25 top-k → RRF fuse. RRF(d) = Σ 1/(k + rank_i(d)) with k=60 (empirically robust per Cormack 2009). RRF is rank-based, so it does NOT need score normalization — common pitfall is min-max-normalizing dense cosine and BM25 raw scores then weighted-summing; BM25 is unbounded and dataset-dependent, breaking the average.

Learned sparse: SPLADE-v3 (lexical-expansion via MLM head, term weights) bridges dense recall and BM25 lexical precision. BGE-M3 produces dense + sparse + multi-vector in one forward pass — single-model hybrid.

Typical fused weighting: 0.6 dense / 0.4 sparse. Fuse top 50 from each; rerank top 30 fused → top 10.

Query transformations

• HyDE: LLM generates a hypothetical answer; embed and retrieve against it. Wins on vocabulary mismatch (medical, legal). Adds 1 LLM call (~300–800 ms small model). Hurts on factual lookup where the generated answer fabricates wrong terminology.
• Multi-query / RAG-Fusion: generate N rewrites, retrieve each, RRF-fuse. Wins on ambiguous queries; cost is N× retriever calls + 1 LLM call.
• Decomposition: break complex query into sub-queries, retrieve per sub, aggregate. Mandatory for multi-hop QA; latency cost compounds.

Production pattern: route via a small classifier — short factual → direct; vocabulary-gap → HyDE; multi-hop → decompose. ≤1B router (or logistic regression on query features) keeps overhead < 50 ms.

Evaluation

• Retrieval-only: NDCG@k (graded relevance, position-discounted), MRR (first-hit reciprocal), Recall@k (coverage), Hit@k (binary). Report at k = 5, 10, 20.
• RAGAS metrics:
  • Faithfulness = claims-extracted-from-answer / claims-supported-by-context (NLI-prompted).
  • Answer relevancy = similarity of answer to LLM-rephrased queries derived from the answer.
  • Context precision (rank-aware: are relevant chunks at top?).
  • Context recall (needs ground truth).
• TruLens RAG triad: context relevance, groundedness, answer relevance — all LLM-as-judge.
• LLM-judge biases: length (verbose answers under-rated 10% by some judges, over-rated by others), positional, self-preference, format. Mitigations: ensemble of 3 judges (+8% accuracy), pairwise prompts with order-swap, calibrated rubrics with anchor examples, capping max_tokens to neutralize length.
• Golden set: 200–500 (query, ideal-answer, supporting-doc-ids) tuples drawn from production logs (sampled, redacted, human-validated). Refresh quarterly.

Hallucination / attribution

• Faithfulness via NLI: split answer into atomic claims (LLM), each claim → (entail / neutral / contradict) over retrieved context. groundedness < 1 → hallucination.
• Citation grounding via Anthropic Citations API (Messages API, citations: {enabled: true}): Claude returns sentence/passage spans into provided documents. Reported case (Endex): source-hallucination 10% → 0%, +20% citations per response. Caveat: citations don't fix factual errors when the source itself is wrong.
• Patronus Lynx, RAGTruth, MiniCheck — open-source hallucination detectors; combine with judge ensemble for cheap online QA.

Index freshness

• Embedding-drift sources: tokenizer change, model version bump, normalization toggle, preprocessing edit, re-embedding only a subset. Rule: one index = exactly one (model_version, pipeline_hash, normalization) tuple. Pin via pipeline_id field on every vector.
• Hot/cold split: hot index — recent N days, frequent incremental upserts; cold index — bulk-rebuilt off-peak (3 AM weekends), promoted atomically. Pinecone, Weaviate, Qdrant support zero-downtime alias swap.
• Re-embed cost: per-token embedding API pricing changes — multiply current rate by token count when budgeting a model swap. OSS local re-embed cost is dominated by GPU-hours, not per-token rates.
• Incremental indexing: emit {doc_id, content_hash, last_modified} events; only re-embed when content_hash changes. Tombstone deletes; never partially update a chunked doc — re-chunk the whole document.

Long context vs RAG

• Long-context wins: small corpora (≤ 200 pages), unique single-document tasks, multi-hop reasoning where chunking would sever dependencies, exploratory analysis.
• RAG wins: large corpora (>1M tokens), low-latency / high-QPS, cost-sensitive, requires citation per answer, frequently updated content.
• "Lost in the middle" / context rot: accuracy degrades for content placed mid-context. Frontier long-context models do well on synthetic NIAH but lose meaningfully on realistic multi-fact recall. Re-check current numbers against your task — published headline scores rarely match production retrieval quality.
• Cost / latency: a full 1M-token request is orders-of-magnitude slower and more expensive per query than RAG at equivalent retrieval quality. Hybrid routing (cheap-RAG-first, escalate-to-long-context) is the dominant pattern.

Multimodal RAG

• ColPali (ICLR 2025): VLM (PaliGemma backbone) → multi-vector embeddings of page-image patches. Late-interaction MaxSim between query token vectors and patch vectors. Eliminates OCR / layout-parse / text-chunking pipelines for PDFs, slides, screenshots.
• ColQwen2 — Qwen2-VL backbone, same recipe; ColSmol — small variant for edge.
• Storage cost is high (multi-vector per page); use binary / scalar quantization, PLAID two-stage filter.
• Use when documents are layout-heavy (financial filings, scientific PDFs, scanned forms).

Failure modes

• Needle-in-haystack: evaluate with NoLiMa / RULER, not synthetic NIAH (saturated).
• Similarity-vs-relevance gap: cosine high, intent wrong (e.g., antonyms cluster nearby in word2vec-derived spaces). Mitigate with reranker + query intent classification.
• Vocabulary mismatch: query uses common terms, docs use jargon (or vice-versa). Mitigate with HyDE or hybrid BM25/SPLADE.
• Multi-hop: single retrieval insufficient; needs decomposition + iterative retrieval (Self-RAG retrieve token, CRAG retrieval-quality evaluator with web fallback).
• Negation / temporal: embeddings ignore "not" and tense; route to a structured filter when detected.

Production patterns

• Caching layers: query-embedding cache (key = hash(query) + model_version), retriever-result cache (TTL ≤ index-rebuild interval), reranker cache (key = hash(query, top_k_doc_ids)), answer cache only if generation is deterministic.
• Retrieval-log mining: sample 1% of queries with retrieved docs and downstream signals (click, thumbs-up, follow-up rephrasing) → weekly golden-set augmentation.
• Drift monitoring: track P50 retrieval-score distribution per day; alert on >2σ shift. Track answer-length distribution and faithfulness on a held-out probe set.
• Self-RAG / CRAG in prod: lightweight retrieval-quality classifier; on "ambiguous" → fall back to web / broader index; on "irrelevant" → skip retrieval.

Authoritative references

• Anthropic Contextual Retrieval
• Anthropic Citations API
• MTEB Leaderboard
• Late Chunking paper, Jina blog
• ColPali paper
• ColBERTv2 paper, Jina-ColBERT-v2
• Self-RAG
• RAGAS metrics, TruLens RAG triad
• Databricks long-context RAG

Guardrails

Before recommending a non-trivial RAG-pipeline change (chunking strategy, embed-model swap, reranker addition, hybrid weights, re-embed):

1. Quote the parameter / approach and any default
2. Cite the originating paper / blog / docs
3. Make the recommendation conditional on observed evidence (NDCG@k, MRR, faithfulness on a golden set, latency P99) — never blanket-tune
4. Verify model versions and pipeline IDs — many features are release-gated and rotate fast.

Tuning without measurement is worse than defaults.