Menú
Design, evaluate, and operate retrieval-augmented generation systems
Instalar con Codex o Claude Copia este prompt, pégalo en Codex, Claude u otro asistente, y deja que revise la página de la skill y la instale por ti.
Basado en la clasificación ocupacional SOC
Build UIs that work for all users including keyboard navigation, screen readers, and WCAG 2.2
Design multi-agent systems with robust tool interfaces, state management, and failure handling
Build ML systems with disciplined training, evaluation, deployment, and safety practices
Design APIs that are stable, ergonomic, and evolvable
Design systems at the right scale with explicit trade-off documentation
Design services that are reliable, observable, secure, and maintainable
| name | rag-system-design |
| description | Design, evaluate, and operate retrieval-augmented generation systems |
| difficulty | staff |
| domains | ["ai-ml"] |
RAG systems fail in ways that are hard to see: wrong chunks retrieved, context window overflows, hallucinations on retrieved content, stale knowledge base. This skill builds RAG systems with rigorous evaluation at every layer.
What questions will users ask? What documents will answer them? What does "good retrieval" look like? Write 50+ question-answer pairs as your evaluation set before writing any code.
Test at least 2 embedding models on your retrieval evaluation set. Measure: retrieval accuracy (does the correct chunk appear in top-k?), latency, cost. Don't default to the most popular model.
Measure:
Target: Recall@5 > 0.80 on your evaluation set before moving to generation.
How do you combine retrieved chunks into a prompt context?
The LLM should cite sources. Implement structured output that includes: answer, confidence, source chunks used. Validate that answers are grounded in retrieved content, not model memory.
Measure:
Use an LLM-as-judge framework (RAGAS or similar) for automated evaluation.
When was the source document last updated? Is the retrieved content stale? Add metadata filtering for date, version, or source. Tell users when information may be outdated.
Track: retrieval latency, retrieval scores over time, user feedback, answer quality. Stale embeddings (from updated documents) will silently degrade quality.
"If retrieval finds relevant documents, the generation will be good" Good retrieval is necessary but not sufficient. LLMs hallucinate on retrieved content, misread it, and fail to synthesize across chunks.
"Chunk size doesn't matter much" Chunk size is the most important RAG hyperparameter. Too small: no context. Too large: diluted signal, context window pressure.
