design-search-index-infra

name	design-search-index-infra
description	Designs full-text and vector search infrastructure — Elasticsearch/OpenSearch mappings and analyzers, vector index parameters (HNSW M/efConstruction, IVF nlist/PQ), BM25+vector hybrid via RRF, offline relevance tuning, capacity/shard topology, and alias-based zero-downtime reindex.
when_to_use	Building or tuning a search backend — defining a text mapping (analyzers/tokenizers/multi-fields), sizing a vector index for recall-vs-latency-vs-memory, fusing lexical and vector into hybrid search, tuning relevance with offline eval, or planning a zero-downtime reindex. NOT for wiring an LLM retrieval/grounding flow (use rag-pipeline) or keeping the index synced from a DB log (use build-cdc-streaming-pipeline).

When to Use

Reach for this when the request is about the search index itself — how documents are mapped, scored, and stored — not the application logic that calls it:

• "Set up a mapping/analyzer so partial-word and stemmed search works"
• "Add autocomplete / typeahead / search-as-you-type"
• "Pick HNSW vs IVF and size M/efConstruction/nlist for N million vectors"
• "Combine keyword (BM25) and semantic (embedding) search into one ranked list"
• "Search relevance is bad — boost titles, add synonyms, tune fuzziness"
• "Reindex 200M docs to a new mapping with no downtime"
• "How many shards/replicas, what refresh interval, how much heap for the HNSW graph?"

NOT this skill:

• Wiring an LLM to answer over the corpus (chunking → embed → retrieve → rerank → ground) → rag-pipeline (this skill builds the index that pipeline queries)
• Keeping the index in sync with a source DB as rows change → build-cdc-streaming-pipeline
• Tuning a relational WHERE/JOIN/GIN query plan in Postgres/MySQL → optimize-sql-query
• Putting a read cache in front of the search cluster → caching-strategy
• Measuring downstream answer quality of an LLM → llm-eval-harness

Steps

1. Classify the query workload first — it dictates index type. Do not vector-index everything.

   Workload	Example query	Index	Scoring
   Exact / filter	status=active, sku=ABC, range, faceting	keyword/numeric, doc_values	none (constant) — wrap in filter (cached, no scoring)
   Full-text relevance	"wireless noise cancelling headphones"	text + analyzer	BM25
   Autocomplete / prefix	"wir" → "wireless…"	search_as_you_type or edge-ngram	prefix match
   Semantic / fuzzy-intent	"thing to block out plane noise"	dense_vector (HNSW)	cosine/dot
   Filtered hybrid	semantic + brand IN (...) + price<200	text + vector + keyword	RRF fusion + filter

   Most real search is the last row. Build all three field families in one index; choose per query, not per cluster.

2. Full-text mapping — be explicit, never rely on dynamic mapping in prod. Disable dynamic or set "dynamic": "strict" so a stray field can't silently become the wrong type. Per field decide: text (analyzed, for relevance) vs keyword (exact, for filter/sort/agg) — you almost always want both via multi-fields:

   {
     "mappings": {
       "dynamic": "strict",
       "properties": {
         "title": {
            
                             
            
                          
                    
                 
                          
           
         
                    
                 
                   
                
                         
                                 
       
     
   
   

Common Errors

• Dynamic mapping in prod. First doc with a stringly-typed number makes the field text; later range queries silently match nothing. Set "dynamic": "strict".
• Wrong distance metric. Indexing cosine-trained embeddings with l2/euclidean returns results — just in the wrong order, with no error. Match similarity to the model.
• Summing BM25 + cosine scores raw. Different scales; one retriever dominates. Use RRF, or min-max normalize each list before weighting.
• Post-filtering vector results. knn top-k then drop non-matching → empty or thin results when matches rank deep. Push the filter into the ANN search; brute-force exact for very selective filters.
• fuzziness: 2 on everything. Matches "cat"→"car"→"can" — precision tanks. Use AUTO (edit distance scaled by term length).
• Edge-ngram with the same analyzer at search time. The query gets shredded into n-grams too, so "wire" matches "fire" via shared grams. Set search_analyzer: standard — index grams, search the whole term.
• HNSW graph that doesn't fit RAM. Once it spills to disk, query latency jumps 10–100×. Compute resident size before indexing; quantize (int8_hnsw) or go IVF-PQ if it won't fit.
• Over-sharding. 500 shards for 10 GB of data — each shard is a Lucene index with fixed overhead; cluster state bloats, GC thrashes. Aim 20–50 GB/shard.
• Reindex with default refresh_interval and replicas≥1. Every batch refreshes + replicates → reindex crawls. Set refresh:-1, replicas:0 during, restore after.
• App pinned to a concrete index name. Any reindex is now downtime + a deploy. Always read/write through an alias from day one.
• Tuning relevance on one query. A title boost that fixes "iphone" can wreck "running shoes review." Gate every change on the offline eval set.

Verify

• Mapping is explicit & immutable-safe: GET <index>/_mapping shows dynamic: strict, every searched field has the intended type/analyzer, and a .raw keyword exists for each sorted/aggregated field.
• Analyzer does what you think: POST <index>/_analyze {"field":"title","text":"running shoes"} emits the expected stemmed/lowercased/ngram tokens (e.g. run, shoe).
• Vector metric & dims match the model: dims equals the embedding model's output, similarity matches its training; a near-duplicate of an indexed doc returns itself as the #1 nearest neighbor.
• Recall measured, not assumed: kNN results compared against an exact brute-force scan on a sample → recall@10 ≥ 0.95 at the chosen ef_search/nprobe; raise the param if below.
• Hybrid beats either alone: on the labeled judgment set, RRF NDCG@10 ≥ max(BM25-only, vector-only), and a query with a hard filter (brand=X) still returns relevant, filter-passing results (no recall cliff, no empty set).
• Relevance change gated: _rank_eval (or offline harness) shows NDCG@10 and recall@k did not regress vs the previous config across all judgment queries.
• Topology sane: shards are 20–50 GB each, heap ≤ 31 GB, _cluster/health is green, and HNSW graphs fit resident RAM (no disk spill in node stats).
• Reindex was truly zero-downtime: alias flipped in a single _aliases call, doc counts reconcile (v2.count == v1.count + writes-during-window), and live search returned 200s with no error spike across the swap.

Done = the index serves the target workload with explicit immutable-safe mapping, measured recall@10 ≥ 0.95 and a non-regressing NDCG@10 on the offline eval set, hybrid+filter returns no empty/cliffed results, and a mapping change can ship via an atomic alias swap with zero search downtime.