deepeval-framework-setup

# DeepEval Framework Setup This skill builds a reusable **LLM-as-judge evaluation harness** around one or more *apps under test* (a chatbot, a RAG pipeline, an AI agent — anything that answers over HTTP). It is the generalized form of `Chapter_19_DeepEval_Framework/03_DeepFramework`, which already evaluates three live targets. Treat that directory as the canonical reference implementation; this skill tells you how to stand up the same thing for a brand-new application. ## When you're invoked, first establish three things 1. **What is the target?** A chatbot, a RAG system, or an agent? How is it reached — local FastAPI, a hosted URL, a function call? What's the request shape and the response shape (JSON vs plain text)? 2. **What does "good" mean here?** Which qualities matter — staying on topic, grounding in retrieved context, no hallucination, no bias/toxicity, no PII leak, citing sources, multi-turn memory? 3. **Which judge LLM?** OpenAI (highest limits), Groq (fast, free tier is rate-limited), or local Ollama (no key, fully offline). Don't guess these. If the user hasn't said, ask — the answers decide the targets, the metric registry rows, and the datasets. ## The mental model ``` app under test ──HTTP──▶ target client ──▶ LLMTestCase ──▶ metric │ judge LLM ◀──────┘ │ score + pass/fail + reason ``` The framework never inspects the model inside the app — it treats the app as a black box reached over HTTP, wraps each response in a DeepEval `LLMTestCase`, and hands it to a metric. The metric prompts the **judge LLM** (a different, trusted model) which returns a numeric score and a written reason. This separation is the whole point: the thing being graded and the grader are different models, so the grade is independent. ## Directory layout to create Mirror this structure. It's deliberately flat so the dashboard and the pytest suites import the *same* modules — never duplicate metric logic between them. ``` <framework>/ ├── .env # keys + JUDGE_PROVIDER (gitignored — never commit) ├── requirements.txt # pinned (see Versions below) ├── conftest.py # loads .env, fixtures, liveness skip, run capture ├── pytest.ini # markers ├── llm_providers/ │ ├── base.py # CompatibleJudge(DeepEvalBaseLLM) │ └── factory.py # get_judge() reads JUDGE_PROVIDER ├── targets/ │ └── <app>.py # HTTP client per app under test ├── datasets/ │ └── <app>_goldens.py # golden Q&A + safety/PII probes ├── dashboard/ │ ├── app.py # FastAPI on :8203 (interactive UI + REST) │ ├── registry.py # the single source of truth: MetricDef rows │ ├── runner.py # run one registry metric end-to-end → JSON │ └── runs_store.py # local run-history capture └── tests/ └── <app>/test_NN_*.py # one pytest file per metric ``` The full, working code for every one of these files lives in `references/templates.md`. Read it when you're writing the files — don't reconstruct them from memory. Read `references/metrics-catalog.md` for the complete list of metrics, their thresholds, and their score directions. ## Setup workflow Follow these in order. Each step is small on purpose — verify before moving on. 1. **Scaffold + venv.** Create the directory tree, a `requirements.txt` with the pinned versions below, then `python3 -m venv venv && venv/bin/pip install -r requirements.txt`. Verify with `venv/bin/python -c "import deepeval, instructor, openai; print(deepeval.__version__)"`. 2. **Judge factory** (`llm_providers/`). Copy `CompatibleJudge` and the factory from `references/templates.md`. One class serves all three providers because OpenAI, Groq, and Ollama all speak the OpenAI-compatible wire protocol; `instructor` handles structured output uniformly. Verify: `JUDGE_PROVIDER=openai venv/bin/python -c "from llm_providers import get_judge; print(get_judge().get_model_name())"`. 3. **Target client(s)** (`targets/`). One small `requests`-based class per app — a `health()`/`is_alive()` and a `chat()` that returns a dataclass. Match the app's real contract: JSON body vs form, JSON response vs plain text, whether it takes conversation `history`. See the two contrasting examples in the templates (local JSON chatbot vs plain-text hosted bot). 4. **Golden dataset(s)** (`datasets/`). A dataclass per golden carrying `input`, `expected_output`, `context` (ground-truth facts), and `categories`. Plus a flat list of adversarial `SAFETY_PROMPTS`. Ground every `expected_output` and `context` in the app's real source of truth — a golden that cites facts the app doesn't have produces a misleading score. 5. **Metric registry** (`dashboard/registry.py`). This is the heart. One `MetricDef` row per (metric × target). Each row names a `factory(judge, threshold)`, a `sample_kind` (which dataset the runner pulls from), a `requires` list (which `LLMTestCase` fields the metric needs), and a `higher_is_better` flag that sets the threshold direction. Adding a metric = adding a row; it appears in the dashboard instantly. 6. **Runner** (`dashboard/runner.py`). Maps a `sample_kind` to a dataset, calls the target, builds the right `LLMTestCase`, runs the metric, returns a JSON-friendly dict. Copy it as-is; extend only the `sample_kind` switch when you add a new kind. 7. **conftest + pytest.ini.** `conftest.py` loads `.env`, exposes `judge`/target/golden fixtures, and auto-skips a suite when its target app is down (`needs_<app>` markers). Register the markers in `pytest.ini` so pytest doesn't warn. 8. **Dashboard** (`dashboard/app.py`). FastAPI on `:8203` that renders straight from the registry — click a metric, see live pass/fail/score/reason, switch judge provider on the fly. Start it with `uvicorn dashboard.app:app --port 8203`. 9. **pytest suites** (`tests/<app>/`). One file per metric, using `assert_test(test_case, [metric])` so `deepeval test run` captures each case (needed for the Confident AI cloud dashboard). Plain `metric.measure()` + `assert is_successful()` works for local pytest but is NOT captured by the CLI. 10. **Verify end-to-end.** Start the target app(s), then run one metric through the dashboard (`curl -X POST localhost:8203/api/run -d '{"metric_id":"...","sample_idx":0}'`) and one pytest suite. Confirm a real score comes back before declaring done. ## Versions — pin these DeepEval's API and packaging move fast; pin to avoid surprises. | Package | Version | Why | |---------|---------|-----| | `deepeval` | **`3.9.9`** | `4.0.6` (latest) ships a **broken `deepeval test run` CLI** — its code does `from deepeval.deepeval.config.settings import ...`, a typo; there is no nested `deepeval/deepeval/` package. The *library* works on 4.x, only the CLI crashes. `3.9.9` is the last release whose CLI works out of the box **and** still has every metric (`PIILeakageMetric`, `KnowledgeRetentionMetric`, `ConversationCompletenessMetric`). | | `openai` | `>=2.0` | judge transport (also used for Groq/Ollama via base_url) | | `groq` | `>=1.0` | optional, for Groq-hosted apps under test | | `instructor` | `>=1.6` | structured output for judge prompts | | `fastapi` / `uvicorn` / `jinja2` | `0.115 / 0.32 / 3.1` | the dashboard | | `pytest` | `>=8` | the suites | Judge models per provider: `openai` → `gpt-4o-mini`, `groq` → `openai/gpt-oss-120b`, `ollama` → `gpt-oss:20b`. Override with `JUDGE_MODEL_OPENAI` / `_GROQ` / `_OLLAMA`. ## Gotchas worth knowing up front These come from actually running the framework — flag them before the user hits them. - **`HallucinationMetric` needs a non-empty `context`.** It throws on an empty list. Out-of-scope / refusal goldens often have no natural context — give them a scope-policy line, or tag them to skip hallucination. - **`.env` is not auto-loaded by the apps under test.** The eval framework's `conftest.py` / dashboard load it, but the chatbot/RAG apps read raw `os.getenv` — export the key (or `set -a; source .env; set +a`) when launching them. - **Groq's free tier caps `gpt-oss-120b` at 8000 TPM.** `deepeval test run` fans out judge calls concurrently and blows past it (429). For large or CLI runs, use the OpenAI judge (`JUDGE_PROVIDER=openai`, `gpt-4o-mini`). - **Score direction is inverted for safety metrics.** Higher is better for quality (threshold = floor); lower is better for hallucination/bias/toxicity/PII (threshold = ceiling). Don't hand-roll the comparison — `metric.is_successful()` already handles the inversion. - **`deepeval test run` only captures `assert_test()` cases.** A manual `measure()` loop runs fine locally but the CLI reports "No test cases found" and pushes nothing to Confident AI. - **Different apps have different contracts.** A local FastAPI bot may return JSON `{reply, model, mode}`; a hosted bot may return plain text with a `visitorId`. The target client absorbs that difference so the rest of the framework doesn't care. ## Adding a new target (a new app to evaluate) Common request: "evaluate my agent the same way." Steps: 1. Add `targets/<app>.py` — a client matching that app's HTTP contract. 2. Add `datasets/<app>_goldens.py` — goldens grounded in that app's truth. 3. Add `<app>` fixtures + a `needs_<app>` liveness skip in `conftest.py`, and the markers in `pytest.ini`. 4. Add `MetricDef` rows with `target="<app>"` to `registry.py` (reuse the metric factories — only the target + dataset change). 5. Extend `runner.py`'s `_call_target` to dispatch to the new client. 6. Add `tests/<app>/test_NN_*.py` suites. Because the registry drives the dashboard, the new target's metrics become clickable buttons with no UI changes. ## Adding a new metric 1. Use a built-in DeepEval metric, or define a `GEval` with a plain-English rubric (best for bespoke criteria like "cites a real source filename"). 2. Write a one-line `factory(judge, threshold)` in `registry.py`. 3. Append a `MetricDef` row (set `higher_is_better` and `sample_kind`). 4. Optionally mirror it as a `tests/<app>/test_NN_*.py` using `assert_test`. That's it — see `references/templates.md` for the exact code shapes.