菜单
Designs full-text and vector search infrastructure — Elasticsearch/OpenSearch mappings and analyzers, vector index parameters (HNSW M/efConstruction, IVF nlist/PQ), BM25+vector hybrid via RRF, offline relevance tuning, capacity/shard topology, and alias-based zero-downtime reindex.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/Sir-chawakorn/sanook-cli --skill design-search-index-infra命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
Gets machine-parseable JSON out of an LLM reliably — prefer provider-enforced grammar (OpenAI `response_format:{type:"json_schema",strict:true}`, Anthropic tool-calling / `tool_choice:{type:"tool"}`, Gemini `responseSchema`) over a "respond in JSON" prompt, define the contract once as a Pydantic/Zod model and validate every response (never trust the string), use constrained decoding / grammars for open weights (Outlines, llguidance, llama.cpp GBNF, vLLM `guided_json`), keep schemas SHALLOW with required fields + enums + `additionalProperties:false`, and build a repair-and-retry loop that handles refusals, `length`/`max_tokens` truncation, and streaming partial JSON (`partial-json-parser`). Schema-guided generation eliminates parse-error retries; validate-and-repair is the backstop, not the primary mechanism.
Builds production data grids that stay fast and accessible at 10k–1M+ rows — decide server-side vs client-side sort/filter/paginate by the dataset-fits-in-memory test (client only under ~10k rows, otherwise push to the API and treat the table as a controlled view of server state), ROW-VIRTUALIZE with TanStack Virtual or react-window so only the visible window mounts (fixed estimateSize, overscan 5–10, measureElement for dynamic rows, contain:strict, and a real scroll container — never table-layout:auto over thousands of rows), build the headless logic with TanStack Table v8 (or AG Grid when you need pinning/grouping/enterprise out of the box), add column resize/reorder/pin, inline edit with optimistic update + rollback on error, row selection with a stable rowId, full keyboard nav with roving tabindex over an ARIA grid (role=grid/row/gridcell, aria-sort, aria-rowcount/aria-rowindex so virtualization stays announced), position:sticky headers, streaming CSV export that doesn't block the main thread, and explici
Builds semantic/vector search — pick an embedding model + dimensionality (and whether to truncate Matryoshka dims) and the matching distance metric (cosine/dot/L2, normalize to unit length so cosine == dot and IP is correct), an ANN index with the recall/latency/memory tradeoff understood (HNSW M/efConstruction/efSearch for low-latency RAM-resident; IVF-PQ nlist/nprobe/PQ for billion-scale compressed; flat/exact for <100k) in pgvector/Qdrant/Milvus/FAISS/Pinecone, chunking + overlap + per-chunk metadata for filtering, HYBRID retrieval fusing BM25 + dense by Reciprocal Rank Fusion (RRF, k≈60) not score addition, a cross-encoder/Cohere reranker over the top-50→k, correct pre-filter-vs-ANN interaction (filterable HNSW, not post-filter that starves k), and offline eval with recall@k / nDCG@10 / MRR against a labeled qrels set. Quantize (scalar/PQ) only after measuring recall loss; tune efSearch/nprobe to a recall target, not a guess.
基于 SOC 职业分类
正在显示 SKILL.md
| name | design-search-index-infra |
| description | Designs full-text and vector search infrastructure — Elasticsearch/OpenSearch mappings and analyzers, vector index parameters (HNSW M/efConstruction, IVF nlist/PQ), BM25+vector hybrid via RRF, offline relevance tuning, capacity/shard topology, and alias-based zero-downtime reindex. |
| when_to_use | Building or tuning a search backend — defining a text mapping (analyzers/tokenizers/multi-fields), sizing a vector index for recall-vs-latency-vs-memory, fusing lexical and vector into hybrid search, tuning relevance with offline eval, or planning a zero-downtime reindex. NOT for wiring an LLM retrieval/grounding flow (use rag-pipeline) or keeping the index synced from a DB log (use build-cdc-streaming-pipeline). |
Reach for this when the request is about the search index itself — how documents are mapped, scored, and stored — not the application logic that calls it:
M/efConstruction/nlist for N million vectors"NOT this skill:
WHERE/JOIN/GIN query plan in Postgres/MySQL → optimize-sql-queryClassify the query workload first — it dictates index type. Do not vector-index everything.
| Workload | Example query | Index | Scoring |
|---|---|---|---|
| Exact / filter | status=active, sku=ABC, range, faceting | keyword/numeric, doc_values | none (constant) — wrap in filter (cached, no scoring) |
| Full-text relevance | "wireless noise cancelling headphones" | text + analyzer | BM25 |
| Autocomplete / prefix | "wir" → "wireless…" | search_as_you_type or edge-ngram | prefix match |
| Semantic / fuzzy-intent | "thing to block out plane noise" | dense_vector (HNSW) | cosine/dot |
| Filtered hybrid | semantic + brand IN (...) + price<200 | text + vector + keyword | RRF fusion + filter |
Most real search is the last row. Build all three field families in one index; choose per query, not per cluster.
Full-text mapping — be explicit, never rely on dynamic mapping in prod. Disable dynamic or set "dynamic": "strict" so a stray field can't silently become the wrong type. Per field decide: text (analyzed, for relevance) vs keyword (exact, for filter/sort/agg) — you almost always want both via multi-fields:
{
"mappings": {
"dynamic": "strict",
"properties": {
"title": {
text; later range queries silently match nothing. Set "dynamic": "strict".l2/euclidean returns results — just in the wrong order, with no error. Match similarity to the model.knn top-k then drop non-matching → empty or thin results when matches rank deep. Push the filter into the ANN search; brute-force exact for very selective filters.fuzziness: 2 on everything. Matches "cat"→"car"→"can" — precision tanks. Use AUTO (edit distance scaled by term length).search_analyzer: standard — index grams, search the whole term.int8_hnsw) or go IVF-PQ if it won't fit.refresh_interval and replicas≥1. Every batch refreshes + replicates → reindex crawls. Set refresh:-1, replicas:0 during, restore after.GET <index>/_mapping shows dynamic: strict, every searched field has the intended type/analyzer, and a .raw keyword exists for each sorted/aggregated field.POST <index>/_analyze {"field":"title","text":"running shoes"} emits the expected stemmed/lowercased/ngram tokens (e.g. run, shoe).dims equals the embedding model's output, similarity matches its training; a near-duplicate of an indexed doc returns itself as the #1 nearest neighbor.ef_search/nprobe; raise the param if below.brand=X) still returns relevant, filter-passing results (no recall cliff, no empty set)._rank_eval (or offline harness) shows NDCG@10 and recall@k did not regress vs the previous config across all judgment queries._cluster/health is green, and HNSW graphs fit resident RAM (no disk spill in node stats)._aliases call, doc counts reconcile (v2.count == v1.count + writes-during-window), and live search returned 200s with no error spike across the swap.Done = the index serves the target workload with explicit immutable-safe mapping, measured recall@10 ≥ 0.95 and a non-regressing NDCG@10 on the offline eval set, hybrid+filter returns no empty/cliffed results, and a mapping change can ship via an atomic alias swap with zero search downtime.
Rules: set search_analyzer ≠ index analyzer for edge-ngram (index n-grams, search the whole query term — otherwise the query is shredded too and precision collapses). Use english/language analyzers for stemming; keep a .raw keyword for anything you sort, aggregate, or exact-match. Set "index": false on fields you only display (saves space). Mapping is immutable — wrong type means reindex (step 7), so get it right now.
Vector index — pick the algorithm by corpus size and recall target. Default HNSW; reach for IVF/PQ only when RAM-bound.
| Param | What it trades | Opinionated default |
|---|---|---|
m (HNSW edges/node) | recall + memory ↑ vs build time | 16 (32 for high-recall >10M) |
ef_construction | build-time recall ↑ vs index speed | 128 (200 if recall short) |
ef_search/num_candidates | query recall ↑ vs latency | 100, raise until recall@10 plateaus |
IVF nlist (partitions) | speed ↑ vs recall | ≈√N vectors |
IVF nprobe (lists scanned) | recall ↑ vs latency | nlist/20, tune up for recall |
| PQ (product quant.) | memory ÷4–16 vs recall ↓ | only when graph won't fit RAM |
~(dims*4 + m*8) bytes × N; a 768-dim, 10M, m=16 index ≈ 31 GB resident. If it won't fit, go IVF-PQ (FAISS/Milvus) or scalar-quantize (int8_hnsw in ES 8.x → ~4× smaller, recall ~unchanged).cosine, or dot_product if you pre-normalize vectors to unit length (skips the per-query magnitude divide → faster). Never l2/euclidean on cosine-trained vectors — silently wrong ranking, not an error.dims must equal the model's output exactly. Truncating/padding to a "round" number breaks similarity.Hybrid — run BM25 and vector separately, then fuse with RRF. Do not just add raw scores. BM25 (unbounded, ~0–30) and cosine (0–1) are different scales; summing lets one dominate. Reciprocal Rank Fusion uses rank position, not score:
rrf_score(d) = Σ_q 1 / (k + rank_q(d)) // k=60 default, sum over each retriever
Use ES rank: { rrf: {...} } / OpenSearch hybrid query, or compute RRF app-side over the two result sets. Weighted fusion (α·norm(bm25) + (1-α)·cosine) works only if you min-max normalize each list first and tune α offline (step 5); RRF needs no normalization and is the safer default.
Filtering is where hybrid breaks. A post-filter (retrieve top-k vectors, then drop ones failing brand/price) can return 0 results when matches sit at rank 5000 — the recall cliff. Push filters into the ANN search (knn.filter in ES, filter clause in Milvus/Qdrant) so the graph traversal only visits passing nodes. For very selective filters (<1% pass), ANN degrades to near-exhaustive — detect it and fall back to a brute-force exact vector scan over the pre-filtered set; it's faster than fighting the graph.
Relevance tuning — change one lever, gate every change on an offline eval set. Never tune by eyeballing one query.
title^3 body^1), synonyms (synonym_graph filter, expand at search time so you can edit without reindex), fuzziness (AUTO = 0 edits <3 chars, 1 <6, 2 else — never blanket fuzziness:2, it wrecks precision), minimum_should_match, phrase/proximity boosts, recency/popularity function_score._rank_eval) or an offline harness. Metrics: recall@k (did we retrieve it at all), P@k, NDCG@10 (rank quality with graded relevance). A change ships only if NDCG/recall doesn't regress on the set — local "looks better" is how you trade one query's win for ten silent losses.Capacity / topology — size shards to data, not to instinct.
shards = ceil(total_primary_GB / 40), round to your data-node count.number_of_replicas: 0 during a bulk reindex, restore to ≥1 after — cuts reindex time ~2×.refresh_interval: default 1s is wasteful for write-heavy/bulk loads. Set 30s (or -1 during pure bulk, then restore) — controls the latency between index and searchability; raise it whenever you don't need sub-second freshness.Lifecycle — alias-based zero-downtime reindex. The app NEVER names a concrete index.
Apps read/write the alias products, which points at products-v1. To change mapping/analyzer/dims:
# 1. create v2 with the NEW mapping, replicas=0, refresh=-1 (fast bulk)
PUT products-v2 { "settings": {"number_of_replicas":0,"refresh_interval":"-1"}, "mappings": {...} }
# 2. backfill v1 -> v2 (async, throttled so you don't starve live traffic)
POST _reindex?wait_for_completion=false { "source":{"index":"products-v1","size":5000},
"dest":{"index":"products-v2"} }
# poll: GET _tasks/<taskId> — bulk in 5–15k-doc batches; size by tuning until throughput plateaus, not by guessing
# 3. restore prod settings, then ATOMIC alias swap (single request = no gap, no double-read)
PUT products-v2/_settings { "number_of_replicas":1, "refresh_interval":"30s" }
POST _aliases { "actions":[ {"remove":{"index":"products-v1","alias":"products"}},
{"add": {"index":"products-v2","alias":"products"}} ]}
# 4. keep v1 until v2 verified in prod, then delete
Catch-up writes during reindex: rows changed after the _reindex snapshot are missed. Either dual-write to both indices during the window, or replay the change log from the snapshot timestamp — the source-of-truth → index sync is owned by build-cdc-streaming-pipeline; this skill only guarantees the swap is atomic.