deepeval-tracing

# DeepEval Tracing Use this skill to instrument an **AI application** — an LLM app, agent, RAG pipeline, or chatbot — with **DeepEval's native tracing** so its execution is visible span by span in **Confident AI's Observatory**. The work is: pick a supported integration when one exists, fall back to manual `@observe` otherwise, give each span a meaningful type, and add tags and metadata. This skill stops at producing well-formed traces. Attaching evaluation metrics and running evals is the `deepeval` skill's job. ## Scope: AI Applications Only Instrument only the AI parts of the system — agent loops and planning, LLM calls, retrieval / vector search, and tool calls. The span types (`llm`, `retriever`, `tool`, `agent`) describe AI components. Do not trace non-AI software (web servers, CRUD backends, infrastructure). If the target has no LLM, agent, retrieval, or tool-calling component, this skill does not apply. ## When to Use vs the `deepeval` and `deepeval-otel` Skills - **This skill (`deepeval-tracing`)** — instrument an app with the DeepEval SDK (`@observe`, framework integrations) so traces reach Confident AI. - **`deepeval` skill** — build pytest eval suites: datasets, metrics, traced evals, `deepeval test run`, iteration. It runs evals *against* an app this skill instrumented. - **`deepeval-otel` skill** — instrument with the vendor-neutral OpenTelemetry SDK instead of the DeepEval SDK (raw OTLP, including non-Python apps). The three are complementary. If unsure between this skill and `deepeval-otel`: use this one when the app is Python and you want the DeepEval SDK; use `deepeval-otel` when you want raw OpenTelemetry or the app is not Python. ## Prerequisites - An AI application in Python with `pip install deepeval`. - For traces to reach Confident AI: `deepeval login`, or an exported `CONFIDENT_API_KEY` (preferred for CI and non-interactive runs). ## Workflow 1. Confirm the target is an AI application (it has LLM calls, an agent loop, retrieval, or tool calls). If it has none of these, stop — this skill does not apply. 2. Detect the framework, model provider, agent SDK, and vector database in use. 3. Read `references/integrations.md` and the exact integration doc for what was detected. Prefer a native integration over manual instrumentation. 4. If no native integration fits, instrument manually with `@observe`. Read `references/tracing.md`. 5. Give each span a meaningful `type` (`llm`, `retriever`, `tool`, `agent`) and capture inputs/outputs. 6. Add trace-level tags and metadata where they help diagnose failure patterns. Never trace secrets, credentials, or raw sensitive data. 7. Confirm `deepeval login` or `CONFIDENT_API_KEY`, then verify traces appear in the Confident AI Observatory. ## Core Principles 1. Instrument AI components only — `llm`, `retriever`, `tool`, `agent` spans. Never trace non-AI software. 2. Prefer a supported integration over manual `@observe`. Manual tracing is the fallback for unsupported frameworks and app-owned wrapper boundaries. 3. Read the exact integration doc before writing tracing code. 4. Give spans meaningful types; let names default to function names unless there is a strong reason to override. 5. Never trace secrets, credentials, API keys, or raw sensitive user data. 6. Producing traces is the scope. Attaching metrics and running evals belong to the `deepeval` skill; raw OpenTelemetry export belongs to `deepeval-otel`. ## References | Topic | File | | --- | --- | | Manual instrumentation: `@observe`, span types, tags, metadata | `references/tracing.md` | | Integration selection rule and framework / model / vector-DB doc index | `references/integrations.md` |