-
Pin the corpus + query shape first. Write down: doc count & avg length, modality (prose / code / tables / mixed), update frequency (static vs hourly), and 10–20 real example questions. Two facts decide everything downstream: (a) are queries lookup ("what is X") or multi-hop ("compare X and Y across docs"), (b) does stale data return wrong answers (→ need re-index or freshness filter). Skip this and you will tune chunk size against questions that don't exist.
-
Chunk by structure, not by character count. Default 400–800 tokens, 10–15% overlap. Split on natural boundaries: prose → headings/paragraphs; code → whole functions/classes (AST-aware, never mid-function — a half function embeds to garbage); markdown → keep a section together, prepend the heading path to each chunk. Tables → keep the header row with every row-group. Store metadata on every chunk: source_path, title, section, chunk_index, updated_at. You will need it for filtering and citations.
-
Embed deterministically and store the contract. Pick one model and freeze it; record model_name, dim, and normalize. Two non-negotiables: (a) embed the query with the exact same model as the docs, (b) if your similarity metric is cosine, L2-normalize both sides. Batch 50–200 chunks/call. Hash each chunk (sha256(text)) and skip re-embedding unchanged chunks on re-index — this is the single biggest cost saver. For code or asymmetric search, prefer an embedding model trained for retrieval (query/doc asymmetry) over a generic sentence model.
-
Choose the store by ops reality, not benchmarks. Already on Postgres → use the pgvector extension (one DB, transactional, metadata filters in SQL). Need managed/huge scale → a hosted vector DB. Prototype/local → an embedded vector store. Index params that actually matter: HNSW m (16–32) and ef_construction (64–200) for build, ef_search at query time for the recall/latency trade. Set the distance metric to match how you embedded (cosine vs inner product vs L2) — a mismatch silently returns wrong neighbors with no error.
-
Retrieve hybrid + rerank, not top-k dense alone. Pull dense (top 20–50) AND keyword/BM25 (top 20–50), fuse with Reciprocal Rank Fusion. Dense misses exact IDs, error codes, rare tokens, function names; BM25 catches them. Apply metadata filters (date, source, type) before scoring, not after, or you lose your top-k to filtered-out rows. Then rerank the fused ~40 candidates with a cross-encoder reranker down to the final 3–8 that go in the prompt. Reranking is usually the highest-leverage relevance win per dollar.
-
Write a grounding prompt that can say "I don't know." Inject retrieved chunks with explicit source labels ([1] path#section). Instruct: answer ONLY from the provided context, cite the [n] you used, and if the context doesn't contain the answer, say so instead of guessing. Put the question after the context. Order chunks best-last if the model shows lost-in-the-middle behavior on long contexts.
-
Measure retrieval and end-to-end as TWO separate numbers. Retrieval: from your example questions, label which chunk(s) are correct, then compute recall@k and MRR — "is the right chunk in the top-k at all." End-to-end: is the final answer correct AND grounded in cited chunks. Diagnosis rule: bad recall@k → fix chunking/embeddings/hybrid (step 2–5); good recall but bad answer → fix the grounding prompt (step 6) or model. Optimizing blindly without splitting these wastes days.
-
Tune cost/latency last, once relevant. Cache query embeddings (same question hits often). Persist doc embeddings — never re-embed on every run. Drop ef_search/top-k until recall@k degrades, then back off one notch. Consider a smaller embedding dim only after confirming recall holds. Add a cheap query-router so trivial lookups skip the reranker.
