LLM and AI testing patterns — mock responses, evaluation with DeepEval/RAGAS, structured output validation, and agentic test patterns (generator, healer, planner). Use when testing AI features, validating LLM outputs, or building evaluation pipelines.
LLM and AI testing patterns — mock responses, evaluation with DeepEval/RAGAS, structured output validation, and agentic test patterns (generator, healer, planner). Use when testing AI features, validating LLM outputs, or building evaluation pipelines.
Patterns and tools for testing LLM integrations, evaluating AI output quality, mocking responses for deterministic CI, and applying agentic test workflows (planner, generator, healer). Of that trio only the healer keeps a local reference here; the planner and generator stages belong to the testing-e2e skill.
DeepEval, RAGAS, VCR.py and Playwright document themselves. This skill carries only the
OrchestKit delta (references/ork-delta.md) plus the house subsets in rules/ and
checklists/. Fetch the source below instead of expecting the material here.
Topic
Source
Full DeepEval metric catalog and per-metric constructor arguments (the house threshold table and the two-metric quick start stay in this file, rules/llm-evaluation.md and checklists/llm-test-checklist.md)
Key rule: NEVER call live LLM APIs in CI. Use mocks for unit tests, VCR.py for integration tests.
DeepEval Quality Quick Start
Validate LLM output quality with multi-dimensional metrics:
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
test_case = LLMTestCase(
input="What is the capital of France?",
actual_output="The capital of France is Paris.",
retrieval_context=["Paris is the capital of France."],
)
assert_test(test_case, [
AnswerRelevancyMetric(threshold=0.7),
FaithfulnessMetric(threshold=0.8),
])
Library notes (DeepEval, RAGAS)
DeepEval metrics expose a reason field alongside the numeric score when include_reason=True, so a failing CI build gets a human-readable explanation without a second LLM call:
metric = AnswerRelevancyMetric(threshold=0.7, include_reason=True)
metric.measure(test_case)
print(metric.score, metric.reason)
# 0.62 "Response addresses the topic but omits the date asked for."
RAGAS uses a class-based metric API — instantiate metric classes and pass an EvaluationDataset. llm= is optional; omit it to use the configured default grader:
from ragas import evaluate
from ragas.metrics import Faithfulness, LLMContextRecall
result = evaluate(
dataset,
metrics=[Faithfulness(), LLMContextRecall()],
)
Bump floors: deepeval >= 4.0, ragas >= 0.4.
House rules the vendor docs do not state (the inverted HallucinationMetric threshold,
the gpt-5-mini grader default, the 95 percent confidence-interval recipe, and the
latency, quality-gate and truncation numbers) are recorded in references/ork-delta.md.
Read that before writing either library's setup code.
Planner: Explores your app and produces Markdown test plans. Owned by the
testing-e2e skill (rules/e2e-ai-agents.md); the CLI and its generated files are
documented at https://playwright.dev/docs/test-agents.
Generator: Converts Markdown specs into Playwright tests, validating selectors
against the running app. Also owned by testing-e2e (rules/e2e-ai-agents.md); the
locator ladder it follows lives in testing-e2erules/e2e-playwright.md.
Healer (references/healer-agent.md): Automatically fixes failing tests by replaying failures, inspecting the DOM, and patching locators/waits. Max 3 healing attempts per test.
Agent initialization is CLI-only (npx playwright init-agents); there is no config key
for it. Only the healing stage keeps a local reference, because its 3-attempt ceiling and
its refusal to touch test logic are house limits rather than vendor defaults.
Edge Cases to Always Test
For every LLM integration, cover these paths:
Empty/null inputs -- empty strings, None values
Long inputs -- truncation behavior near token limits
Timeouts -- fail-open vs fail-closed behavior
Schema violations -- invalid structured output
Prompt injection -- adversarial input resistance
Unicode -- non-ASCII characters in prompts and responses
See checklists/llm-test-checklist.md for the complete checklist.
Anti-Patterns
Anti-Pattern
Correct Approach
Live LLM calls in CI
Mock for unit, VCR for integration
Random seeds
Fixed seeds or mocked responses
Single metric evaluation
3-5 quality dimensions
No timeout handling
Always set < 1s timeout in tests
Hardcoded API keys
Environment variables, filtered in VCR
Asserting only is not None
Schema validation + quality metrics
Related Skills
ork:testing-unit — Unit testing fundamentals, AAA pattern
ork:testing-integration — Integration testing for AI pipelines
