| name | ais-rag |
| description | RAG pipeline production — ingestion, chunking chiến lược, embedding, hybrid search, rerank. Không index trùng, search dưới 200ms. |
ais-rag — RAG Pipeline
Files
| File | Mục đích |
|---|
templates/chunking.py | chunk_document() — markdown/code/text splitters with overlap |
templates/embedding.py | EmbedCache (SQLite) + embed_batch() with OpenAI |
templates/schema.sql | pgvector DDL — documents table, IVFFlat + GIN indexes |
templates/hybrid-search.py | hybrid_search() — vector cosine + keyword ts_rank combined |
templates/rerank.py | LLM-based reranking with RERANK_PROMPT |
Usage
from templates.chunking import chunk_document
chunks = chunk_document(text, "markdown")
Performance targets: search < 200ms, rerank top-3 < 1s.
Deduplication: hash content before insert to avoid duplicates.
Anti-Rationalization
| Excuse | Reality |
|---|
| "Hybrid search không cần, vector đủ" | Keyword miss → 0 results cho thuật ngữ chính xác |
| "Chunk overlap 0 là đủ" | Câu bị cắt giữa chừng → context incomplete |
| "pgvector index sau, data ít" | Full scan khi có 10k+ docs → search > 1s |
Red Flags
- Chunk overlap = 0
- Chỉ dùng vector search, không hybrid
- pgvector không index IVFFlat
Verification
