Enhancement-overlay SOP for adding sparse (BM25 / keyword) retrieval alongside
dense (embedding) retrieval. Activate when a calling agent is building, reviewing,
or debugging a retrieval pipeline whose corpus contains exact-match tokens —
identifiers, error codes, SKUs, API/function names, proper nouns, citations, rare
jargon — that pure dense embedding silently misses. Encodes the single decision
rule (**hybrid is traffic-driven, not theoretical: add sparse only when the query
share that depends on exact tokens is non-trivial**), the wiring of
QueryFusionRetriever-style fusion (RRF vs alpha-weighted), and per-query-type alpha
tuning. Frame the work as recovering lexical identity that dense pooling destroys,
not as "add keyword search for completeness". Cross-links [[llamaindex]].
Instrucciones de origen · Vista previa de solo lectura
name
agentsop-hybrid-retrieval
version
0.1.0
description
Enhancement-overlay SOP for adding sparse (BM25 / keyword) retrieval alongside
dense (embedding) retrieval. Activate when a calling agent is building, reviewing,
or debugging a retrieval pipeline whose corpus contains exact-match tokens —
identifiers, error codes, SKUs, API/function names, proper nouns, citations, rare
jargon — that pure dense embedding silently misses. Encodes the single decision
rule (**hybrid is traffic-driven, not theoretical: add sparse only when the query
share that depends on exact tokens is non-trivial**), the wiring of
QueryFusionRetriever-style fusion (RRF vs alpha-weighted), and per-query-type alpha
tuning. Frame the work as recovering lexical identity that dense pooling destroys,
not as "add keyword search for completeness". Cross-links [[llamaindex]].
["the corpus contains exact-match tokens: identifiers, error codes, SKUs, part numbers, API/function names, proper nouns, legal/medical citations, rare jargon","users report 'I searched the exact code/name and got nothing / the wrong doc' on a dense-only retriever","reviewing a retriever where traffic includes lexical-identity lookups but only embeddings are wired","tuning recall on a corpus where both meaning AND exact strings matter","deciding between pure dense, hybrid, or sparse-first for a new RAG corpus"]
when_not_to_use
["traffic is purely semantic / conceptual with <5% lexical-identity queries — hybrid is over-engineering","the corpus has no stable identifiers and queries never reference exact strings","pre-baseline: ship dense-only and measure first; hybrid is a Stage-3 optimization (see [[llamaindex]])"]
Hybrid Retrieval · Dense + Sparse SOP
Third-person operating model for a coder agent that owns retrieval recall on a
corpus where both meaning and exact tokens matter. The audience is the LLM
agent writing or reviewing retrieval code — not an end user.
One sentence: Dense captures meaning, sparse captures exact tokens; hybrid
wins when both matter — but only fuse them when traffic actually carries
exact-match queries, and tune the blend per query type or hybrid loses to dense.
1. 何时激活 (Activation Rules)
Activate this skill when any of the following holds:
The corpus contains exact-match tokens that a query may reference verbatim:
error codes (ERR_SSL_PROTOCOL), SKUs / part numbers (A1-2293-X), API or
function names (as_query_engine), proper nouns, legal/medical citations
(42 U.S.C. § 1983), version strings, rare jargon, ticket IDs.
A bug report says "I searched the exact code/name/string and got nothing", or
"the right document exists but dense retrieval ranks it below fuzzy near-misses".
PR review surfaces a retriever serving lexical-identity traffic but wired
dense-only (index.as_retriever(...) / similarity_search(...) with no
sparse leg).
You are tuning recall and have already exhausted the cheap dense knobs (prompt,
embedding model, chunk size) per the [[llamaindex]] optimization ladder — hybrid
is the next rung.
The user mentions hybrid search, BM25, sparse retrieval, RRF, QueryFusionRetriever,
EnsembleRetriever, or vector-store-native hybrid (Qdrant/Weaviate/Pinecone).
Do not activate when:
Traffic is purely semantic ("what does X mean?", "summarize the policy") with a
lexical-identity share <5% — adding BM25 doubles index footprint for no gain.
The corpus has no stable identifiers and no query ever quotes an exact string.
No dense baseline + eval loop exists yet. Hybrid is a Stage-3 optimization
([[llamaindex]] Stage 3 step 4): baseline and measure before fusing.
2. 核心心智模型 (Core Mental Model)
Three principles. Violating any of them is why teams "try hybrid and conclude it
didn't help".
Principle 1 — Dense and sparse fail in opposite directions
Dense embedding models destroy lexical identity by pooling token representations:
querying a specific error string yields a vector that captures rather than . BM25 does the
inverse — it scores against an and is blind to
synonyms and paraphrase. (TianPan, , 2026; cited in
[[llamaindex]] Dilemma 2.)
"document about SSL
errors"
"document containing this exact string"
inverted index of exact tokens
Hybrid search in production
Operational corollary: the symptom "I pasted the exact code and got nothing"
is not a bug in the embedding model — it is the embedding model working as
designed. The fix is a second retriever that indexes tokens, not a better
embedding.
Principle 2 — Hybrid is traffic-driven, not theoretical
Whether to add sparse is decided by the query-type distribution of real traffic,
not by a belief that "more retrievers = better". The decision threshold is the
lexical share: the fraction of queries whose correct answer hinges on an exact
token. Below ~5% → dense-only. 5–50% → hybrid. Above ~50% (code, logs, legal) →
invert to sparse-first with dense as a rerank signal. (Cited in [[llamaindex]]
OP-04 / Dilemma 2.)
Principle 3 — One global alpha underperforms; tune per query type
alpha is the dense↔sparse blend (alpha=1 → pure dense, alpha=0 → pure sparse).
A semantic query wants high alpha; a lexical-identity query wants low alpha. A single
global alpha picked to help lexical queries hurts the semantic slice — which is
exactly why a flat alpha "loses to pure dense" and teams wrongly conclude hybrid
failed. Tune alpha per query type, or route per type and pick alpha per route.
(LlamaIndex alpha-tuning blog; [[llamaindex]] Dilemma 2 结果.)
The fusion method (RRF vs weighted) and the fusion parameter (alpha) are two
separate decisions. RRF is score-scale-robust and parameter-light; weighted fusion
is tunable but requires score normalization. Choosing neither deliberately is the
third silent failure.
3. SOP 工作流 (Agentic Protocol)
Four stages. Each gates the next. This overlay assumes a working dense baseline +
eval loop already exists (see [[llamaindex]] Stages 1–2); do not start here.
Stage 0 — Decide if the corpus needs sparse at all
Sample ≥50 real queries (or representative synthetic ones if pre-launch).
Label each: semantic ("what does X mean?") / lexical ("find error code
ABC-123", "the function named foo_bar") / mixed.
Compute the lexical share:
< 5% → dense-only. Stop; hybrid is over-engineering. Record the decision.
5–50% → add hybrid (Stages 1–3).
> 50% (code search, log search, legal citation lookup) → invert: BM25-first,
dense as a fallback / rerank signal.
Confirm the corpus actually carries the tokens queries reference (an identifier
that never appears verbatim in any chunk can't be recovered by BM25 either).
Artifact: a 3-line decision note co-located with the retriever recording the lexical
share and the chosen shape.
Stage 1 — Wire BM25 + dense as two retrievers
from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.retrievers.bm25 import BM25Retriever
dense = index.as_retriever(similarity_top_k=10) # embedding leg
sparse = BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=10)
Hard rules:
Both legs index the same node set. A BM25 leg built over a different/stale node
set silently drops recall (it can only return tokens it has).
Retrieve wider per leg (top_k 10–20 each) than the final cut — fusion needs
candidates to combine. Narrow after fusion (and after any reranker).
BM25 needs the raw nodes/corpus, not just the vector store — persist or rebuild it
alongside the index, or use a vector store with native hybrid (Stage 1-alt).
Stage 1-alt — vector-store-native hybrid. If the vector store does sparse
internally (Qdrant sparse_vectors, Weaviate hybrid(alpha=...), Pinecone sparse-dense,
pgvector + ts_rank), prefer it: one round-trip, one consistency domain, no separate
BM25 index to keep in sync. Use the framework fusion path only when the store has no
native hybrid.
Stage 2 — Fuse (RRF or alpha-weighted)
Pick the fusion method deliberately (OP-02):
# Reciprocal Rank Fusion — score-scale-robust, parameter-light. Good default.
fused = QueryFusionRetriever(
[dense, sparse],
mode="reciprocal_rerank", # RRF: combine by rank, ignore raw score scales
similarity_top_k=8,
num_queries=1, # set >1 to also fan out query rewrites
use_async=True,
)
# Relative-score / weighted fusion — tunable blend; requires score normalization.
fused = QueryFusionRetriever(
[dense, sparse],
mode="relative_score",
retriever_weights=[0.6, 0.4], # ~ alpha=0.6 toward dense
similarity_top_k=8,
)
Default to RRF unless you have a labeled set to tune weights on — RRF sidesteps
the dense-cosine vs BM25-score scale mismatch that breaks naive weighted sums.
Stage 3 — Tune alpha by query type (the gate)
Build a small labeled set per query type (semantic / lexical / mixed) — reuse the
Stage-0 sample.
Evaluate alpha (or retriever_weights) at {0.0, 0.25, 0.5, 0.75, 1.0} on each
type separately, scoring recall / MRR / hit-rate per type.
Expect: lexical queries peak near low alpha (sparse-leaning), semantic near high
alpha (dense-leaning), mixed in between.
If a single global alpha cannot satisfy both types without regressing one →
route by query type and apply per-route alpha (hand off classification to
[[agentsop-query-routing]] if present), or split into two retrievers selected per query.
Gate: hybrid must lift the lexical slice with no regression on the semantic
slice vs the dense baseline. If it regresses semantic, the alpha is wrong, not
hybrid.
The most common false negative: tuning one global alpha, watching semantic recall
drop, and reverting to dense. The correct read is "alpha was global; tune per type".
4. 操作模型 (Operation Models)
Format: Trigger / Action / Output / Evidence.
OP-01 WhenHybridChecklist
Trigger: Deciding whether to add sparse to a dense pipeline.
Action: Run the Stage-0 lexical-share checklist. Confirm: (a) corpus has
exact-match tokens, (b) traffic references them, (c) lexical share ≥5%, (d) those
tokens appear verbatim in chunks. All four must hold.
Output: A go/no-go with the lexical share recorded; "no" is a valid, common
outcome.
Evidence: [[llamaindex]] Dilemma 2 决策步骤; OP-04 ("trigger is traffic-driven,
not theoretical").
OP-02 ChooseFusionMethod
Trigger: Two legs wired; need to combine their result lists.
Action: Default RRF (mode="reciprocal_rerank") — rank-based, robust to the
dense-cosine vs BM25-score scale gap, no weight to tune. Use weighted /
relative-score only when you have a labeled set to fit weights and have normalized
scores. Never naive-sum un-normalized scores.
Output: One deliberate fusion method + the reason it was chosen.
Trigger: Hybrid approved; sparse leg not yet built.
Action: Build BM25Retriever.from_defaults(nodes=...) over the same node
set as the dense index; persist/rebuild it as a deployment artifact alongside the
vector index. Set per-leg similarity_top_k wide (10–20).
Output: A sparse retriever consistent with the dense index, returning enough
candidates to fuse.
Trigger: The vector store supports sparse internally.
Action: Use the store's native hybrid (Qdrant sparse vectors / Prefetch +
fusion, Weaviate hybrid(query, alpha=...), Pinecone sparse-dense vectors, pgvector
full-text + vector) instead of a separate BM25 index. One round-trip, one
consistency domain.
Output: Hybrid with no second index to keep in sync; lower operational surface.
Trigger: Fusion wired; recall not yet optimized; or hybrid "loses to dense".
Action: Evaluate alpha / weights at {0, 0.25, 0.5, 0.75, 1.0} on labeled
subsets per query type, not globally. Lexical → low alpha, semantic → high.
Output: Per-type alpha curve; the alpha that lifts lexical without regressing
semantic.
Evidence: LlamaIndex alpha-tuning in hybrid search blog; [[llamaindex]]
Dilemma 2 ("tune per type, not globally — otherwise hybrid underperforms dense").
OP-06 RouteThenAlpha
Trigger: No single global alpha satisfies both query types without regression.
Action: Classify the query type first, then dispatch to a retriever configured
with the per-type alpha (or pure dense / pure sparse). Hand classification to a
router ([[agentsop-query-routing]] if available).
Output: Each query gets its optimal blend; semantic and lexical slices both peak.
Evidence: [[llamaindex]] Dilemma 2 结果 (per-type tuning); routing as the
mechanism to apply it.
Action: Make BM25 the primary retriever; use dense as a fallback / rerank
signal to catch paraphrase, not as the lead leg. Equivalent to alpha pinned low.
Output: Recall dominated by exact-token matching where that is what users want;
dense recovers the semantic minority.
Evidence: [[llamaindex]] Dilemma 2 step 4 ("BM25-first, dense as fallback rerank
signal"); TianPan production write-up.
OP-08 HybridEvalGate
Trigger: Before merging any hybrid change.
Action: Run the eval loop on both slices: assert lexical-slice recall/MRR
rises vs dense baseline AND semantic-slice metrics do not regress. Gate merge on
both. A lexical lift bought with a semantic regression is not a win.
Output: Quantitative proof hybrid helped where intended and harmed nothing else.
Dilemma 1 — Hybrid (BM25 + dense) vs pure dense: worth the complexity?
困境: Dense is the modern default; BM25 looks like "the old keyword thing". Adding
hybrid doubles the index footprint, adds a BM25 index to keep in sync, and introduces
alpha tuning. Is the complexity justified, or is a better embedding model enough?
约束:
Pure dense silently fails on exact identifiers, code, error strings, SKUs, API
names, rare jargon (TianPan 2026, quoted in Principle 1) — and a better embedding
model does not fix it; lexical loss is a property of pooled representations.
Hybrid adds a second index, a fusion step, and a tuning parameter — real ops cost.
决策步骤:
Build the query-type taxonomy from real traffic: semantic / lexical / mixed
(Stage 0).
Lexical share <5% → dense-only; the complexity is not justified.
Evaluate alpha at {0, 0.25, 0.5, 0.75, 1.0} on labeled per-type subsets.
结果: Hybrid lifts the lexical slice with no degradation on the semantic
slice — if alpha is tuned per type. A single global alpha often loses to pure dense
on semantic queries, which is why teams sometimes wrongly conclude "hybrid didn't
help". The decision is traffic-driven, not theoretical. (Verbatim from
[[llamaindex]] Dilemma 2.)
可提取的操作: OP-01, OP-05, OP-07, OP-08.
Dilemma 2 — Alpha for keyword-heavy vs semantic queries: one knob, two demands
困境: After wiring hybrid, a single alpha must serve both a user pasting
ERR_TLS_CERT_INVALID (wants exact-token match, low alpha) and a user asking "why is
my connection failing?" (wants meaning, high alpha). Picking alpha=0.5 helps neither
fully; picking alpha to win lexical regresses semantic, and vice versa.
约束:
alpha=1 = pure dense, alpha=0 = pure sparse; the optimum differs by query
type, not by corpus.
Choosing alpha to maximize average recall across mixed traffic can land in a valley
that underperforms pure dense on the semantic majority — the classic "hybrid hurt
us" report.
RRF reduces but does not eliminate the tension: it still implicitly weights the two
rankings.
决策步骤:
Split the labeled set by query type (semantic / lexical / mixed).
Sweep alpha per type; observe lexical peaks low, semantic peaks high.
If one global alpha exists that lifts lexical with zero semantic regression →
pin it (cheapest).
Else route by query type and apply per-route alpha / pure-leg selection
(OP-06) — classify first, blend second.
For >50% lexical corpora, skip the balancing act: invert to BM25-first (OP-07).
结果: Per-type alpha (or per-type routing) makes both slices peak simultaneously.
The error to avoid is treating alpha as a single global hyperparameter — that is the
documented cause of "hybrid underperforms dense". (LlamaIndex alpha-tuning blog;
[[llamaindex]] Dilemma 2.)
可提取的操作: OP-05, OP-06, OP-08.
6. 反模式与边界 (Anti-patterns & Boundaries)
Top anti-patterns (instant red flags in code review)
#
Anti-pattern
Why it's wrong
Correct move
A1
Adding hybrid by default on a corpus that is purely semantic
Doubles index footprint and ops surface for no recall gain; <5% lexical traffic
OP-01: gate on lexical share; dense-only is the right answer for semantic corpora
B2 No dense baseline + eval loop yet — baseline first; hybrid is a Stage-3
optimization, not a starting point ([[llamaindex]] Stage 3).
B3 The recall problem is actually chunking, embedding-model mismatch, or missing
metadata filters — fix the cheaper knob first ([[llamaindex]] optimization ladder).
B4 The exact tokens users query don't appear verbatim in any chunk — BM25 can't
recover what isn't indexed; fix ingestion, not retrieval.
B5 Top-1 is wrong but the right doc is in top-k — that's a reranking problem,
not a recall problem ([[llamaindex]] OP-03 AddReranker).
PR review smells
QueryFusionRetriever([...]) with no mode= set and no comment on fusion choice.
A single hard-coded alpha / retriever_weights with no per-type eval behind it.
BM25Retriever.from_defaults(nodes=other_nodes) where other_nodes differs from
the dense index's node set.
Per-leg similarity_top_k equal to the final desired count (no headroom for fusion).
Hybrid added but the eval suite only reports a single aggregate recall number.
Hybrid hand-rolled with dense_scores + bm25_scores summed without normalization.
7. 跨框架对照 (Cross-Framework Reference Table)
How "dense + sparse, fused" looks across the common stacks. Cross-links [[llamaindex]].
7.1 LlamaIndex — QueryFusionRetriever
from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.retrievers.bm25 import BM25Retriever
dense = index.as_retriever(similarity_top_k=10)
sparse = BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=10)
fused = QueryFusionRetriever(
[dense, sparse],
mode="reciprocal_rerank", # or "relative_score" / "dist_based_score"
retriever_weights=[0.6, 0.4], # used by weighted modes; ~alpha toward dense
similarity_top_k=8,
num_queries=1, # >1 also fans out query rewrites
)
Fusion modes: reciprocal_rerank (RRF, default-recommended), relative_score,
dist_based_score (the latter two are weighted). ([[llamaindex]] OP-04.)
EnsembleRetriever fuses with Reciprocal Rank Fusion under the hood; weights
biases the RRF contribution per retriever (the LangChain analogue of alpha).
7.3 Raw RRF (framework-agnostic)
defrrf(result_lists, k=60, top_k=8):
scores = {}
for results in result_lists: # each = ranked list of doc idsfor rank, doc_id inenumerate(results):
scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (k + rank + 1)
returnsorted(scores, key=scores.get, reverse=True)[:top_k]
fused = rrf([dense_ids, bm25_ids]) # rank-based, no score normalization
RRF combines by rank, so dense-cosine and BM25 raw scores never need to share a
scale — the reason it is the safe default (OP-02). k≈60 is the canonical constant
(Cormack et al., 2009).
tsvector full-text + vector distance, combined in SQL
hand-weighted in the ORDER BY expression
Milvus
hybrid search with WeightedRanker / RRFRanker
ranker choice + weights
Prefer native hybrid when available (OP-04): one round-trip, one consistency domain,
no separate BM25 index to keep in sync. Weaviate's alpha is the literal parameter the
[[llamaindex]] alpha-tuning blog generalizes.
