deepeval-framework-setup
Set up a DeepEval LLM-as-judge evaluation framework from scratch for any chatbot, RAG pipeline, AI agent, or LLM-backed app under test — the same architecture used in Chapter 19 (ShopSphere chatbot, RAG Explorer, and the live BrowserBash bot). Use this skill WHENEVER the user wants to "evaluate", "test", "score", "benchmark", "add metrics to", "measure quality of", or "QA" a chatbot / RAG / agent / LLM app, or asks to "set up DeepEval", "build an eval harness", "judge an LLM", "add a new eval target", "add a metric", or replicate the Chapter 19 framework for a new application — even if they don't say the word "DeepEval". Covers the judge factory (OpenAI/Groq/Ollama), HTTP target clients, the metric registry, golden datasets, the FastAPI dashboard, the pytest suites, version pins, and the known gotchas.
Source facts
- Repository
- PramodDutta/AITesterBlueprint2x
- Last source activity
- June 27, 2026 at 05:05
- Detected SKILL.md language
- English
- Stars
- 29
- Forks
- 41
Install options
The review-first prompt is selected by default. You can switch to a direct command or download a local copy.
Review the source files
Read SKILL.md and any companion files shown by SkillsMP before deciding whether to install.
Showing SKILL.md
| name | deepeval-framework-setup |
|---|---|
| description | Set up a DeepEval LLM-as-judge evaluation framework from scratch for any chatbot, RAG pipeline, AI agent, or LLM-backed app under test — the same architecture used in Chapter 19 (ShopSphere chatbot, RAG Explorer, and the live BrowserBash bot). Use this skill WHENEVER the user wants to "evaluate", "test", "score", "benchmark", "add metrics to", "measure quality of", or "QA" a chatbot / RAG / agent / LLM app, or asks to "set up DeepEval", "build an eval harness", "judge an LLM", "add a new eval target", "add a metric", or replicate the Chapter 19 framework for a new application — even if they don't say the word "DeepEval". Covers the judge factory (OpenAI/Groq/Ollama), HTTP target clients, the metric registry, golden datasets, the FastAPI dashboard, the pytest suites, version pins, and the known gotchas. |