rag-and-vector-search

name	rag-and-vector-search
description	Use when building RAG systems, implementing semantic/hybrid search, selecting vector databases, tuning retrieval quality, or choosing chunking and embedding strategies.

RAG and Vector Search

Embedding Model Selection

Model	Dims	Best For
text-embedding-3-large	3072	Highest accuracy (OpenAI); supports Matryoshka dim reduction
text-embedding-3-small	1536	Cost-effective default (OpenAI)
voyage-3	1024	Code, legal, finance domains (best retrieval quality)
gte-Qwen2-7B-instruct	3584	Best open-source; instruction-tuned
bge-large-en-v1.5	1024	Strong open-source English, smaller footprint
all-MiniLM-L6-v2	384	Fast/lightweight, prototyping
multilingual-e5-large	1024	Multi-language (requires query/passage prefixes)

Matryoshka Embeddings

Models like text-embedding-3-large support dimension reduction: truncate vectors to 256/512/1024 dims with minimal quality loss. Reduces storage 3-12x. Test recall at target dimension before committing.

Never mix embedding models in the same index -- vectors from different models are incompatible.

Chunking Decisions

Strategy	When
Token-based (512-1000)	Default; predictable size
Semantic/header-based	Markdown/structured docs; preserves logical units
Recursive character	Unstructured text; LangChain default
Parent-child	Need small chunks for retrieval precision, large for LLM context

• Chunk size: 500-1000 tokens default; smaller for precision, larger for context
• Overlap: 10-20% to avoid losing boundary context
• Always test chunk size impact on retrieval quality for your specific corpus

Distance Metrics

Metric	When
Cosine	Default; works with normalized embeddings
Dot Product	When magnitude carries meaning
Euclidean (L2)	Raw/unnormalized embeddings

Index Selection by Scale

Vector Count	Index Type	Notes
< 10K	Flat (exact)	No approximation needed
10K-1M	HNSW	Good recall/speed tradeoff
1M-100M	HNSW + INT8 quantization	Reduces memory ~4x
> 100M	IVF + PQ or DiskANN	Trades recall for scale

HNSW Tuning

Scale	M	efConstruction	efSearch (95% recall)	efSearch (99% recall)
< 100K	16	100	64	128
< 1M	32	200	128	256
> 1M	48	256	128	256

Higher M = better recall but more memory. Memory per vector: dimensions * bytes_per_dim + M * 2 * 4 bytes.

Vector Database Selection

DB	Strength	Best For
pgvector	Already using Postgres; hybrid FTS+vector	Small-medium scale, simplicity
Qdrant	Filtering, quantization, Rust perf	Production workloads needing metadata filters
Weaviate	GraphQL API, multi-modal, hybrid built-in	Multi-modal search, rapid prototyping
Pinecone	Fully managed, zero ops	Teams without infra capacity
Turbopuffer	S3-backed, cost-effective at scale	Large-scale with cold storage economics
Elasticsearch 8.x	Existing ES stack; native RRF	Hybrid search with mature text search

Retrieval Architecture

Hybrid Search (Preferred for Production)

Combine dense (vector) + sparse (BM25/FTS) retrieval. Two fusion approaches:

• RRF (Reciprocal Rank Fusion): Works well without tuning, robust default. Score = sum of 1/(k + rank) across result lists, k=60.
• Linear combination: More control but requires tuning alpha. Normalize scores before combining.

Reranking (Always Worth It)

• Retrieve 20-50 candidates with hybrid search
• Rerank with cross-encoder (e.g., cross-encoder/ms-marco-MiniLM-L-6-v2)
• Cohere Rerank API: managed option, supports rerank-english-v3.0 and multilingual
• ColBERT / late-interaction: token-level matching, better for long documents than bi-encoder reranking
• Return top 3-5 to LLM
• For diversity: use MMR (lambda_mult=0.5 balances relevance vs diversity)

pgvector + FTS Pattern

Store embeddings and tsvector in same table. Use CTE with RRF to combine vector similarity rank and text search rank in a single query.

Advanced RAG Patterns

GraphRAG

Build knowledge graph from documents, then traverse graph relationships during retrieval. Best for corpora with rich entity relationships (legal, biomedical, enterprise docs). Use with Neo4j or networkx.

Contextual Retrieval (Anthropic Pattern)

Prepend chunk-specific context before embedding: "This chunk is from section X of document Y and discusses Z." Improves retrieval by 20-67% on benchmarks. Compute once at index time.

Proposition-Based Chunking

Decompose documents into atomic propositions ("The Eiffel Tower is in Paris", "It was completed in 1889") instead of fixed-size chunks. Better precision for fact-lookup tasks. Higher indexing cost.

Late Chunking

Embed the full document first (using long-context model), then pool token embeddings into chunks. Preserves cross-chunk context that gets lost with chunk-then-embed.

RAG Pipeline Opinions

Retrieval

• Multi-query retrieval: generate 3-5 query variations for better recall on ambiguous questions
• Parent document retriever: index small chunks, return parent context to LLM
• Contextual compression: extract only relevant portions from retrieved docs before sending to LLM
• Metadata filtering: always index source, timestamp, category; filter at query time to reduce noise

Generation

• Always include citation markers ([1], [2]) in prompt template
• Ask for confidence score; instruct model to say "I don't have enough information" when context is insufficient
• Evaluate groundedness: NLI-based check that response is entailed by retrieved context

Evaluation Metrics

• Retrieval: Precision@K, Recall@K, MRR, NDCG
• Generation: Groundedness (NLI), faithfulness, answer relevance
• Test retrieval and generation independently; don't just evaluate end-to-end

Quantization Tradeoffs

Type	Size Reduction	Recall Impact	When
FP16	2x	Negligible	Default for GPU
INT8 scalar	4x	< 1% loss	Production default
Product Quantization	16-32x	2-5% loss	Memory-constrained, > 100M vectors
Binary	32x	Significant	First-pass candidate filtering only

Memory Estimation

total_bytes = num_vectors * (dimensions * bytes_per_dim + M * 2 * 4)

Example: 1M vectors, 1536 dims, FP32, M=16 = ~6.1 GB vectors + ~128 MB index overhead.

Cross-References

• ai-ml:llm-application-patterns -- prompt engineering, agent patterns, production deployment
• ai-ml:structured-output-patterns -- extracting structured data from retrieved documents
• ai-ml:embedding-and-representation-learning -- embedding models, fine-tuning for retrieval