| name | trace-based-testing |
| description | Use this to turn real production traces into a regression test suite, so bugs you saw once never come back. Trigger on "turn traces into tests", "capture this bug as a test", "build tests from production data", "my eval set is out of date", or after debugging a production incident. This is the loop that connects observability to evaluation. |
| license | CC0-1.0 |
Trace-based testing
Observability records what really happened; evaluation checks whether it was good. Trace-based testing is the bridge: promote real production traces into repeatable test cases. Done consistently, your eval set stays representative on its own and every incident makes the app more robust.
The loop
- A trace shows a problem (a bad answer, a hallucination, a tool misfire, a slow/expensive run). You find it via alerts (
set-up-drift-alerts) or debugging (debug-agent-from-traces).
- Capture the trace's input as a test case. The input, the retrieved context, the expected-correct output (write the gold answer, or a rubric). Redact PII first (
redact-pii-for-tracing).
- Add it to the eval set (
build-eval-dataset), tagged with the incident so you know why it exists.
- Fix the bug, confirm the new case passes.
- CI runs it forever (
eval-driven-development) so the bug cannot silently return.
Make it systematic, not manual
- Sample continuously, not just on incidents. Periodically pull a stratified sample of production traces (by topic, difficulty, outcome) and fold them into the eval set so it tracks the real input distribution as it drifts.
- Auto-flag candidates: traces with low online-eval scores, thumbs-down feedback, errors, or high cost are prime candidates to promote to test cases.
- Keep a "replay" path: re-run a captured input through the current app and diff against the recorded/expected output. Most eval frameworks (promptfoo, DeepEval) support running a dataset of captured cases directly.
What to capture per trace
Input, retrieved context / tool results, the model + prompt version at the time, the actual output, and the expected output (your judgment of what good looks like). The first four come from the trace; the last one is the human/label step.
Verify
- A real production failure is now a test case that fails on the old code and passes on the fixed code.
- The eval set grows over time and reflects current production inputs.
- Feedback (thumbs-down) and low-score traces get triaged into candidate test cases regularly.
Anti-patterns
- Fixing a production bug without capturing it as a test (you will debug it again).
- An eval set frozen at launch while production inputs drift away from it.
- Replaying raw traces with PII into a test suite or third-party judge (redact first).
- Capturing the input but not a clear expected output (then the "test" cannot pass or fail meaningfully).
Grounding
This operationalizes the observability-to-eval feedback loop: online quality signals (LLM-as-a-judge, Zheng et al. 2023, arXiv:2306.05685) and human feedback surface weak traces; those become evaluation cases (see Your AI Product Needs Evals). Ties together debug-agent-from-traces, build-eval-dataset, and eval-driven-development.
