| name | dd-check-telemetry |
| description | Verify Datadog telemetry is flowing correctly using MCP tools |
| tools | ["file_read"] |
Verify Telemetry
Confirm that a running demo stack is sending telemetry to Datadog. Uses the Datadog MCP server — no fallback.
Instrumentation-mode differences
Read .d-scribe.json for the instrumentation field. When OTel mode is in use, service identity is emitted via OpenTelemetry resource attributes instead of Datadog tracer tags:
| Concept | datadog / ddot | otel |
|---|
| Service name | service:<name> | service.name:<name> |
| Environment | env:<env> | deployment.environment:<env> |
| Version | version:<v> | service.version:<v> |
When querying the Datadog API for APM spans or LLM Obs traces, use the appropriate tag key for the project's mode. The Datadog intake normalizes both, but pipelines that filter on service: will miss OTel-emitted spans unless they also filter on service.name:.
Prerequisites
- Demo stack running (
docker compose up -d)
- Datadog MCP server configured and accessible
.env populated with valid DD_API_KEY and DD_SITE
Workflow
Step 1: Check MCP availability
Attempt to list services using the Datadog MCP tools. If MCP tools are not available, stop and inform the user:
Datadog MCP server is required for telemetry verification.
Setup: https://github.com/DataDog/datadog-mcp-server
Do not proceed without MCP — partial checks would give false confidence.
Step 2: Read project config
Extract from project files:
docker-compose.yml → list of application service names (exclude datadog-agent, traffic, postgresql, redis, keycloak)
.env → DD_ENV value (used to scope all queries)
AGENTS.md → active features (check for mentions of "Database Monitoring", "RUM", "Profiling")
- Note whether a frontend service exists
Step 3: Check service registration
For each application service, query the Datadog service catalog via MCP.
Report: service found with correct env tag, or FAIL with diagnostic hint.
Step 4: Check logs
For each service, search recent logs (last 15 minutes) via MCP. Build the query filter based on .d-scribe.json instrumentation:
- datadog|ddot →
service:<svc> env:<dd_env>
- otel →
service.name:<svc> deployment.environment:<dd_env>
Verify:
- Logs exist (count > 0)
- At least one log contains
dd.trace_id (trace-log correlation is working)
Report: log count + correlation status, or FAIL with diagnostic hint.
Step 5: Check traces
Search recent traces/spans via MCP. Build the query filter based on .d-scribe.json instrumentation:
- datadog|ddot →
service:<svc>
- otel →
service.name:<svc>
Verify:
- Spans exist for each service
- At least one trace contains spans from 2+ different services (distributed tracing works)
Report: trace count + multi-service trace found, or FAIL with diagnostic hint.
Step 6: Check correlation
Perform cross-signal verification:
- Log-to-trace: Pick a trace_id from a log entry, fetch that trace via MCP, confirm it exists
- Trace-to-log: Pick a trace_id from a span, search logs for that trace_id, confirm correlated logs exist
- RUM-to-backend (if frontend active): Search RUM events via MCP, check they carry backend trace_ids
- DBM (if Database Monitoring active): Search for database query spans with DBM metadata
Report each check as PASS or FAIL with diagnostic hint.
Step 7: Report
Present a structured checklist grouping all results:
Service Registration
[PASS] api-gateway (env: my-demo-260331)
[PASS] user-service
...
Logs
[PASS] api-gateway — 142 logs, trace correlation OK
...
Traces
[PASS] Distributed trace found: api-gateway → user-service → project-service
Correlation
[PASS] Log-to-trace roundtrip
[PASS] Trace-to-log roundtrip
...
For each FAIL, include the diagnostic hint from references/diagnostic-hints.md.
End with a summary: "X/Y checks passed. [All clear / N issues to fix]"
Constraints
- Readonly — never modify the project, docker-compose, or Datadog configuration
- MCP-only — use Datadog MCP tools exclusively
- Stop cleanly if MCP is unavailable
- Idempotent — can be run repeatedly as the SE fixes issues