rag-cascading-search

name	rag-cascading-search
description	Production RAG with hybrid retrieval (vector + keyword) + LLM reranking + provenance metadata, for question-answering over private documents.
when_to_use	Building Q&A over private corpus with target p95 < 200ms; need citations + grounding; corpus size 10k–10M chunks.
applies_to	["ai-system","agent-product"]
sources	["https://docs.pinecone.io/docs/cascading-search","https://github.com/anthropics/skills/data"]

Pattern

Two-stage retrieval with LLM-as-judge reranking:

User query
  ↓
Stage 1: vector + keyword hybrid search (k=50)
  • Pinecone dense embedding (cosine sim)
  • BM25 sparse keyword match
  • Reciprocal-rank fusion to merge
  ↓
Stage 2: LLM rerank top 50 → top 5
  • Claude Haiku 4.5 with rerank prompt
  • Score each chunk on relevance + faithfulness
  ↓
Build context with source provenance
  ↓
Answer generation (Claude Sonnet 4.6)
  ↓
Citation enforcement: every claim → [ref:chunk_id]

Why

• Pure vector search: misses literal-match queries ("what's the SLA in section 4.2?")
• Pure keyword: misses semantic queries ("how do we handle outages?")
• No reranking: top-5 from 50 has 30% noise; LLM rerank cuts to <5%
• No provenance: hallucinations undetectable
• Single-stage with LLM: too expensive at scale; reranking is cheaper than full generation

How

1. Index with both embeddings + sparse vectors

from pinecone import Pinecone
pc = Pinecone()
index = pc.Index("docs-prod")

# Cascading-search embedding model (Pinecone-hosted)
for chunk in chunks:
    index.upsert(records=[{
        "id": chunk.id,
        "_text": chunk.text,             # field for embedding
        "metadata": {
            "source": chunk.source_url,
            "page": chunk.page,
            "doc_title": chunk.doc_title,
            "indexed_at": chunk.indexed_at,
        }
    }])

2. Hybrid retrieve

results = index.search(
    namespace=user_id,
    query={"top_k": 50, "inputs": {"text": user_query}},
)

3. LLM rerank

rerank_prompt = f"""
Given the user query and 50 candidate chunks, score each on relevance (0-10).
Return JSON: [{{"chunk_id": "...", "score": N, "reason": "..."}}, ...]

Query: {user_query}
Chunks: {chunks_json}
"""

reranked = call_claude_haiku(rerank_prompt)
top_5 = sorted(reranked, key=lambda x: x["score"], reverse=True)[:5]

4. Generate with citation enforcement

System prompt requires [ref:chunk_id] after every factual claim. EVAL suite tests this (see EVAL-citation.md template in great_cto).

Anti-patterns

• Reranking with same model as generation: wastes budget. Use Haiku for rerank, Sonnet for generation.
• Re-embedding on every query: cache normalized query embeddings if same query repeats.
• Storing PII in metadata: chunk metadata is returned to LLM context — same exfiltration surface as the chunk content. Filter aggressively.
• No user_id namespace (multi-tenant): cross-user retrieval = isolation bug. Pinecone namespaces are the cheap fix.
• No staleness check: indexed_at should be in metadata; LLM can mention "last updated YYYY-MM-DD" instead of pretending knowledge is current.

Cost (rough, for 10k queries/day)

Item	Cost/day
Pinecone hybrid search × 10k	~$1.50
Haiku rerank × 10k (50 chunks × ~200 tokens × $0.0008/1k)	~$8
Sonnet generation × 10k (5 chunks × ~400 tokens out × $0.015/1k)	~$30
Total	~$40/day ~$1200/mo

Cap (monthly-budget-llm-usd: 1500 in PROJECT.md) → 80% headroom.

Eval coverage

This skill expects these EVAL-*.md files in tests/eval/:

• EVAL-citation.md — every claim has [ref:...]
• EVAL-rag-poisoning.md — malicious instruction in indexed doc, model refuses
• EVAL-staleness.md — model mentions doc indexed_at when serving stale content
• EVAL-cross-user-isolation.md — User A's chunks never appear in User B's response