deepeval-llm-evaluation

name	DeepEval LLM Evaluation
description	Test LLM applications with DeepEval, pytest-style unit tests for LLM outputs using G-Eval, answer relevancy, faithfulness, hallucination and custom metrics, with CI quality gates and dataset-driven regression runs.
version	1.0.0
author	thetestingacademy
license	MIT
tags	["deepeval","llm-evals","g-eval","hallucination","answer-relevancy","faithfulness","llm-testing","pytest","ci-gates"]
testingTypes	["llm-evals","unit","regression"]
frameworks	["deepeval","pytest"]
languages	["python"]
domains	["ai","llm","api"]
agents	["claude-code","cursor","github-copilot","windsurf","codex","aider","continue","cline","zed","bolt","gemini-cli","amp"]

DeepEval LLM Evaluation Skill

You are an expert AI quality engineer specializing in DeepEval. When the user asks you to test, evaluate, or gate LLM application outputs, follow these instructions.

Core Principles

1. Evals are unit tests. Write them pytest-style, run them in CI, fail builds on regressions. No dashboard-only quality.
2. Metric per failure mode. Pick metrics for the failures that matter (hallucination, irrelevance, unfaithfulness to context), not every metric available.
3. Thresholds are contracts. Every metric gets an explicit threshold agreed with the team; a metric without a threshold is a vibe.
4. Datasets over ad-hoc prompts. Evaluate against a versioned golden dataset, grow it from production failures.
5. LLM-as-judge needs spot checks. Periodically hand-verify judge scores; recalibrate criteria when the judge drifts from human judgment.

Setup

pip install deepeval
# judge model key (defaults to OpenAI; other providers configurable)
export OPENAI_API_KEY=sk-...
deepeval login   # optional: Confident AI dashboard for run history

Project Structure

llm-app/
├── evals/
│   ├── conftest.py            # fixtures: app client, dataset loader
│   ├── datasets/
│   │   └── golden_v3.jsonl    # versioned eval cases
│   ├── test_correctness.py    # G-Eval correctness suite
│   ├── test_rag_quality.py    # faithfulness + relevancy for RAG
│   └── test_safety.py         # hallucination, bias, toxicity
└── .github/workflows/evals.yml

Writing Eval Tests

import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    AnswerRelevancyMetric,
    FaithfulnessMetric,
    HallucinationMetric,
    GEval,
)
from deepeval.test_case import LLMTestCaseParams

def make_case(query: str) -> LLMTestCase:
    response = my_app.answer(query)          # your application under test
     LLMTestCase(
        =query,
        actual_output=response.text,
        retrieval_context=response.chunks,   
    )

 ():
     = make_case()
    assert_test(, [AnswerRelevancyMetric(threshold=)])

 ():
     = make_case()
    assert_test(, [FaithfulnessMetric(threshold=)])


correctness = GEval(
    name=,
    criteria=,
    evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
    threshold=,
)

 ():
     = LLMTestCase(
        =,
        actual_output=my_app.answer().text,
        expected_output=,
    )
    assert_test(, [correctness])