aiase-rag

name	aiase-rag
description	RAG (Retrieval-Augmented Generation) architecture — naive RAG pipeline, advanced techniques (HyDE, reranking, compression), modular RAG, GraphRAG, and practical debugging checklist. Load when the user asks about RAG, vector databases, embeddings, chunking, or retrieval systems from the AIASE course.

RAG Architecture

From AIASE 2026 (NCKU), Week 6. Practical patterns for building, debugging, and scaling retrieval-augmented generation systems.

Naive RAG: Three-Step Pipeline

# Indexing (offline, one-time)
chunks = split_text(doc, chunk_size=200, overlap=40)
embeddings = [embed(c) for c in chunks]
vectorstore.add(documents=chunks, embeddings=embeddings)

# Querying (online, per-request)
context = vectorstore.query(embed(user_query), k=3)
prompt = build_prompt(user_query, context)
answer = llm.generate(prompt)

Critical Parameters

Parameter	Recommended Range	Rationale
chunk_size	150–300 chars	Balances context vs. semantic purity
chunk_overlap	20% of chunk_size	Prevents info loss at boundaries
top_k	3–5	Balances recall vs. API cost/latency
embedding_dim	1536 (text-embedding-3-small)	384 for smaller/faster models

Five Limitations of Naive RAG

1. Retrieval noise — semantic similarity ≠ factual relevance
2. Context window overflow — too many chunks exhaust the window
3. Redundant information — similar chunks fetched, wasting tokens
4. Multi-hop reasoning failure — can't chain across documents
5. Generation ignores retrieved context — model falls back on parametric knowledge

Advanced Techniques

Pre-Retrieval

Technique	Mechanism	When to Use
Query Expansion	Generate alternative phrasings of the query	Query is ambiguous or short
HyDE	Generate a hypothetical answer, use as retrieval seed	Improves recall for factual questions
Multi-query	Decompose complex questions into sub-queries	Multi-part or multi-hop questions

Post-Retrieval

Technique	Mechanism	Tool
Re-ranking	Cross-encoder reorders Top-K by true relevance	Cohere Rerank, cross-encoders
Context Compression	Retain only query-relevant sentences	LLMLingua
Filtering	Remove chunks below relevance threshold	Custom scoring

Specialized Systems

System	Key Idea	Best For
Self-RAG	Model emits reflection tokens ([Retrieve], [IsREL], [IsSUP]) to self-control retrieval	Dynamic retrieval decisions
FLARE	Generate answer → retrieve only on uncertainty tokens	Reducing unnecessary retrieval
GraphRAG	Knowledge graph + community detection for corpus-level reasoning	Multi-hop across large corpora
HippoRAG	Simulates hippocampal 3-layer memory via Personalized PageRank	Long-term associative recall

Modular RAG (Current Best Practice)

Pluggable components arranged in arbitrary topology:

• Search, Memory, Fusion, Route, Predict modules
• Enables parallel retrieval streams
• Mix different retrieval strategies per query type

Debugging Checklist

Symptom	Fix
Irrelevant results	Reduce chunk_size
Lost cross-segment info	Increase chunk_overlap
Missed relevant passages	Increase top_k
LLM invents facts not in context	Enforce "respond with 'insufficient knowledge'" in system prompt
High latency	Reduce top_k, add filtering before reranker
High cost	Switch from long-context to RAG (25,000× cheaper than 1M-token context)

Cost: RAG vs. Long Context

Basis	Figure	Context
Single query (1M-token context vs. RAG)	~1,250× more expensive	Per-request comparison; RAG retrieves ~1K tokens instead
At scale (many queries, warm cache)	~25,000× more expensive	Bulk workloads; RAG cache hits compound savings

Both figures come from AIASE W6 course material. The 20× gap between them reflects cache amortization and retrieval reuse across queries.

Rule: Use RAG unless latency is critical OR corpus is <100K tokens.

Recommended Tools

Category	Tools
Frameworks	LangChain, LlamaIndex
Vector Stores	ChromaDB (dev), Qdrant, Weaviate (prod)
Embeddings	BGE, E5 (multilingual SOTA); text-embedding-3-small (OpenAI)
Re-rankers	Cohere Rerank, cross-encoder
Evaluation	RAGAs (context precision, answer faithfulness metrics)

RAGAs Evaluation Metrics

• Context Precision — are retrieved chunks actually relevant?
• Context Recall — are all relevant chunks being retrieved?
• Answer Faithfulness — does the generated answer stick to retrieved context?
• Answer Relevance — does the answer address the question?

---

See also: [[aiase-token-economics]] for cost tradeoffs, [[aiase-agent-fundamentals]] for integrating RAG into agent workflows.