datarobot-external-agent-monitoring

name	datarobot-external-agent-monitoring
description	Instrument any external or existing AI agent with OpenTelemetry to send traces, logs, and metrics to DataRobot for monitoring, observability, and governance. Use when the user says "add tracing/observability/monitoring to my agent", wants to instrument an existing agent project in their IDE, or wants to send agent traces, logs, or metrics to DataRobot.

DataRobot External Agent Monitoring Skill

This skill helps you instrument any AI agent — regardless of framework or deployment environment — to send OpenTelemetry telemetry (traces, logs, metrics) to DataRobot. It also creates a shell deployment in DataRobot as the telemetry routing target.

Quick Start

Most common use case: Instrument an existing agent project, regardless of whether it was built on DataRobot or elsewhere, with DataRobot monitoring

1. The user invokes the skill from inside their project — typically: "Add tracing to my agent"
2. The skill resolves the target project (current IDE workspace / working directory if no path is given), then detects the framework and any existing OTel setup
3. It resolves a Use Case as the telemetry target (asks for the user's Use Case ID, or offers to create one), generates instrumentation code, and wires it in
4. The agent sends traces, logs, and metrics to DataRobot, where they appear under the Use Case's Tracing tab

Examples:

• "Add tracing to my agent" (resolves to the current workspace)
• "Instrument my agent in ./my_agent for DataRobot monitoring"

When to use this skill

Use this skill when an existing DataRobot user has built an agent elsewhere and wants to bring it in for monitoring. Specifically:

• Bring an externally-built (brownfield) agent into DataRobot for monitoring under a Use Case
• Add OpenTelemetry tracing to an agent project
• Send agent traces, logs, and metrics to DataRobot
• Instrument a Google ADK, LangChain, LangGraph, CrewAI, LlamaIndex, PydanticAI, or any Python agent

Supported Frameworks

Framework	Detection	OTel Strategy
Google ADK	google-adk in deps or google.adk in imports	Lazy trace injection via callback (ADK overwrites TracerProvider)
LangChain / LangGraph	langchain or langgraph in deps/imports	Auto-instrumentor + standard setup
CrewAI	crewai in deps/imports	Auto-instrumentor + standard setup
LlamaIndex	llama-index or llama_index in deps/imports	Auto-instrumentor + standard setup
PydanticAI	pydantic-ai or pydantic_ai in deps/imports	Standard setup + required Agent.instrument_all() (instrumentation is opt-in)
Generic Python	None of the above detected	Manual span instrumentation

Workflow

Follow these steps in order. Present the plan to the user and wait for approval before executing.

Step 1: Detect & Analyze

1. Read the project's dependency file (requirements.txt, pyproject.toml, setup.py, poetry.lock, or uv.lock)
2. Scan Python source files for framework imports
3. Check for existing OTel setup (look for opentelemetry imports, existing TracerProvider/LoggerProvider/MeterProvider configuration)
4. Identify the framework using the detection table above
5. Read the corresponding framework reference file from the frameworks/ directory next to this SKILL.md:
   • Google ADK → frameworks/google-adk.md
   • LangChain/LangGraph → frameworks/langchain-langgraph.md
   • CrewAI → frameworks/crewai.md
   • LlamaIndex → frameworks/llamaindex.md
   • PydanticAI → frameworks/pydantic-ai.md
   • Generic Python → frameworks/generic-python.md

Step 2: Check Prerequisites

1. Ensure DATAROBOT_API_TOKEN is available without having the user paste it into chat (a pasted token would be logged in the transcript). Check the environment and the project .env. If the token is missing, create or update a project .env file with the DataRobot variables and have the user paste their Personal API key into that file directly (in their editor); read it from there. Ensure .env is gitignored. This skill targets existing DataRobot users: create a Personal API key at <your DataRobot URL>/account/developer-tools (Personal API keys tab; see the datarobot-setup skill). (No DataRobot account at all? https://www.datarobot.com/trial/.)
2. Check if DATAROBOT_ENDPOINT env var is set. If not, ask the user (default: https://app.datarobot.com/api/v2).
3. Derive DATAROBOT_OTEL_ENDPOINT automatically: if DATAROBOT_ENDPOINT ends with /api/v2, strip it and append /otel (e.g., https://app.datarobot.com/api/v2 → https://app.datarobot.com/otel).
4. Determine the telemetry target (Use Case) — this is the primary entity, and works the same whether the agent was built on DataRobot or elsewhere. Only collect the choice here; do not run any script or create/validate anything yet — that happens once in Step 4, after the user approves the plan (running it here risks creating a Use Case the user never approved, and a duplicate when Step 4 runs).
   • Ask the user for their Use Case ID. DataRobot users typically already organize work in a Use Case.
   • If they don't have one (a brand-new or externally-built project), offer to create one. Ask only for a name; the description is auto-generated.
   • Record the choice (existing Use Case ID, or the name for a new one) to use in Step 4. The create_use_case.py helper will resolve it to an entity ID of the form experiment_container-<use_case_id> at execution time.
5. Check if the datarobot Python SDK is available. If not, install it: pip install datarobot.
6. Check if OTel packages are already in the project's dependencies.

Security note: Never ask the user to paste an API token into chat, and never echo tokens or .env contents into transcripts or logs. Collect the token only via the project .env file (the user edits the file directly) and read it from there; keep .env gitignored. If credentials are accidentally exposed, rotate them immediately.

Step 3: Present Plan

Tell the user what you detected and present the changes you will make:

• Framework detected (or generic Python)
• Existing OTel setup found (if any)
• New dependencies to add
• New files to create (dr_otel_config.py, and optionally dr_agent_metrics.py for frameworks with custom metrics)
• Existing files to modify (agent entrypoint, dependency file)
• Telemetry target: enter an existing Use Case ID, or if user does not have one, generate a net new Use Case container and ID for user. Only list a shell deployment in the plan if the user explicitly asked for deployment-level monitoring; if they chose a Use Case, do not mention or ask about a deployment.

Wait for user approval before executing. If the user has already given explicit consent to implement or deploy, that counts as approval — no need to re-ask.

Step 4: Execute

1. Add dependencies to the project's dependency file:

   • opentelemetry-sdk
   • opentelemetry-api
   • opentelemetry-exporter-otlp-proto-http
   • Framework-specific packages (see framework reference file)

2. Generate dr_otel_config.py using the generic pattern below, adapted per the framework reference file.

3. Wire into agent entrypoint: Add import and call to configure_otel() at startup. Follow the framework reference file for specific wiring instructions (auto-instrumentors, callbacks, etc.).

4. Generate dr_agent_metrics.py if the framework reference file specifies custom metrics callbacks.

5. Resolve the Use Case telemetry target (primary entity). This is the only place the helper script runs — once, here, using the choice collected in Step 2 (never during prerequisites). Validate the user's existing Use Case, or create a net new one if they have none:

   set -a; source .env; set +a   # load DATAROBOT_API_TOKEN etc. from .env (not the command line)
   # Existing Use Case:
   python <skill_scripts_dir>/create_use_case.py --use-case-id <use_case_id>
   # No Use Case yet — create one (name only; description auto-generated):
   python <skill_scripts_dir>/create_use_case.py --name "<project_name> Monitoring"
   

   It returns entity_id as experiment_container-<use_case_id> — this is the OTel entity used at runtime.

6. (Optional) Create shell deployment — only if the user explicitly asks for deployment-level monitoring (drift, etc.). If the user chose a Use Case as the target, do not ask about or prompt for a deployment ID — the Use Case is the complete target on its own. Skip this step entirely unless the user raised it themselves.

   python <skill_scripts_dir>/create_shell_deployment.py \
     --name "<project_name> Monitoring" \
     --description "OTel telemetry sink for <framework> agent"
   

   The script automatically enables prediction row storage and automatic association ID generation on the deployment. If created, its deployment-<id> entity can be used as the target instead of the Use Case.

7. Report results: Write the resolved non-secret runtime vars into the project .env — never print the token. Confirm the Use Case ID (and deployment ID, if created):

   # appended to .env (DATAROBOT_API_TOKEN already present there; do not echo it):
   DATAROBOT_ENTITY_ID=experiment_container-<use_case_id>
   DATAROBOT_OTEL_ENDPOINT=<otel_endpoint>
   

Step 5: Verify & Provide Runtime Instructions

1. Optionally run the verification script (loads credentials from .env; don't put the token on the command line):

   set -a; source .env; set +a
   python <skill_scripts_dir>/verify_otel_connection.py
   

2. Provide the user with the env vars to set in their runtime environment:

   • DATAROBOT_API_TOKEN — DataRobot API key
   • DATAROBOT_ENTITY_ID — experiment_container-<use_case_id> (Use Case target; or deployment-<id> if a shell deployment was created instead)
   • DATAROBOT_OTEL_ENDPOINT — {DATAROBOT_ENDPOINT}/otel

3. Explain how to view the telemetry. For a Use Case target, use the dr CLI's xp plugin (works in a local terminal or DataRobot Codespaces); this is the view_command returned by create_use_case.py:

   dr plugin install xp                                   # one-time
   dr xp --entity-id <use_case_id> --enable-logs --enable-metrics
   #     ^ the BARE use_case_id, NOT the experiment_container- prefixed form
   

   Then open the local panel at http://127.0.0.1:8090. You'll see:

   • Tracing: Span hierarchy (agent orchestration, LLM calls, tool calls)
   • Logs: Structured logs correlated with traces via traceId
   • Metrics: Custom metrics (request count, latency, LLM calls, tool calls)

Generic OTel Configuration Pattern

Generate a dr_otel_config.py with a configure_otel() function that the project calls at startup, before any agent code runs. The full annotated template lives in reference/dr_otel_config.md — read it before generating code. Framework-specific files in frameworks/ layer additional setup on top.

Critical rules:

1. Always pass endpoint= and headers= directly to exporters — NEVER use OTEL_EXPORTER_OTLP_* env vars (some frameworks detect these and create conflicting providers)
2. Be additive — add DataRobot as an additional span processor to any existing TracerProvider, don't replace it
3. Use SimpleSpanProcessor (not Batch) to avoid flush-before-shutdown issues
4. Use DELTA temporality for metrics (required by DataRobot)

Provider initialization order: some frameworks override the global TracerProvider at startup (notably Google ADK), which drops the DataRobot exporter. The additive pattern and per-framework workarounds (e.g. lazy injection via callbacks) are covered in reference/dr_otel_config.md and the framework reference files — always check them.

DataRobot Tracing Table — Span Attribute Mapping

DataRobot's tracing UI (Data Exploration > Traces) maps specific span attributes to table columns. Using the correct attribute names is critical for data to appear in the dashboard.

Column Mapping

Tracing Table Column	Span Attribute	Aggregation Rule
Prompt	gen_ai.prompt	First span with this attribute wins
Completion	gen_ai.completion	Last span with this attribute wins
Tools	tool_name	Lists all unique values across all spans in the trace
Cost	datarobot.moderation.cost	Summed across all spans in the trace

Important: DataRobot looks for tool_name (underscore), NOT tool.name (dot). Some frameworks (e.g., LangGraph) do not set tool_name by default — you must add it manually as a span attribute inside each tool call.

All Recognized Span Attributes

Attribute	Description	Example
gen_ai.prompt	User input / prompt text	"Analyze policy XYZ"
gen_ai.completion	Model output / response	"Policy matched..."
gen_ai.request.model	Model used for the call	"gpt-4o"
gen_ai.usage.prompt_tokens	Input token count	150
gen_ai.usage.completion_tokens	Output token count	320
tool_name	Name of tool/function called (required for Tools column)	"search_database"
tool.parameters	Tool call parameters (JSON string)	'{"query": "..."}'
datarobot.moderation.cost	Cost of this span (summed for trace total)	0.0023

Helper Scripts

create_use_case.py

Resolves the primary telemetry target: validates an existing Use Case, or creates a net new one when the user has none.

# Existing Use Case:
python <scripts_dir>/create_use_case.py --use-case-id <use_case_id>
# Create new (name only; description auto-generated):
python <scripts_dir>/create_use_case.py --name "My Agent Monitoring"

Requires env vars: DATAROBOT_API_TOKEN, DATAROBOT_ENDPOINT

Returns JSON:

{
  "use_case_id": "6123abc",
  "entity_id": "experiment_container-6123abc",
  "otel_endpoint": "https://app.datarobot.com/otel",
  "view_command": "dr xp --entity-id 6123abc --enable-logs --enable-metrics"
}

create_shell_deployment.py

Optional. Creates a shell deployment in DataRobot as a telemetry routing target, for users who also want deployment-level monitoring.

python <scripts_dir>/create_shell_deployment.py \
  --name "My Agent Monitoring" \
  --description "OTel telemetry sink for my agent"

Requires env vars: DATAROBOT_API_TOKEN, DATAROBOT_ENDPOINT

Returns JSON:

{
  "deployment_id": "abc123",
  "entity_id": "deployment-abc123",
  "otel_endpoint": "https://app.datarobot.com/otel"
}

verify_otel_connection.py

Sends test telemetry to verify the OTel pipeline is working.

python <scripts_dir>/verify_otel_connection.py

Requires env vars: DATAROBOT_API_TOKEN, DATAROBOT_ENTITY_ID, DATAROBOT_OTEL_ENDPOINT

Returns JSON:

{
  "status": "success",
  "traces": "sent",
  "logs": "sent",
  "metrics": "sent"
}

Dependencies

Required for instrumentation (added to user's project):

opentelemetry-sdk
opentelemetry-api
opentelemetry-exporter-otlp-proto-http

Required for shell deployment creation (available in the skill's script environment):

datarobot

Best practices

1. Call configure_otel() before any agent/framework initialization — some frameworks capture the provider at import time
2. Never set OTEL_EXPORTER_OTLP_* env vars — pass endpoint and headers directly to exporters to avoid conflicts
3. Use SimpleSpanProcessor over BatchSpanProcessor — avoids flush issues on short-lived processes
4. DELTA temporality for metrics — DataRobot requires delta aggregation for counters and histograms
5. Check framework reference files for initialization order issues before generating code

Error handling

Common errors and solutions:

Error	Cause	Solution
Traces not appearing in DataRobot	Framework overwrites TracerProvider	Use lazy injection pattern (see framework reference)
401 Unauthorized from OTel endpoint	Invalid API token	Verify DATAROBOT_API_TOKEN is correct
404 from OTel endpoint	Wrong endpoint URL	Ensure DATAROBOT_OTEL_ENDPOINT ends with /otel
Metrics not appearing	OTEL_EXPORTER_OTLP_* env vars set	Remove env vars, use direct exporter config
DATAROBOT_ENTITY_ID format error	Missing entity-type prefix	Must be experiment_container-<use_case_id> (Use Case) or deployment-<id>, not just <id>

Resources

• DataRobot Model Monitoring Documentation
• OpenTelemetry Python SDK
• OpenTelemetry OTLP Exporter