datadog

name	datadog
description	Query Datadog metrics, events, APM, and monitors via pup CLI. USE WHEN checking golden signals, querying service dashboards, analyzing APM data, or investigating monitor alerts.
metadata	{"triggers":["datadog","metrics","spans","traces","pup"],"provides":["metrics-query","timeseries-query","event-search","apm-query","monitor-query"],"requires":["pup-cli","DD_API_KEY","DD_APP_KEY"]}

Datadog Skills (via pup CLI)

Datadog access uses the pup CLI — a Go binary with 200+ commands across 33 Datadog products. All commands run in terminal and output JSON by default.

Local Setup

On macOS, install pup with:

brew install datadog/tap/pup

Authentication

pup authenticates via:

1. API keys (DD_API_KEY + DD_APP_KEY env vars) — preferred, works everywhere
2. OAuth (pup auth login) — fallback for codespaces when API keys are not set

Verify Auth

pup auth test    # validates connectivity
pup version # should be 0.45.1+

Agent Mode (Structured JSON Output)

FORCE_AGENT_MODE=1 is set globally in the devcontainer and CCA workflows. This makes pup return structured JSON for --help and auto-approve confirmation prompts — optimized for machine consumption.

⛔ All new code invoking pup MUST ensure agent mode is active. Either:

• Rely on the FORCE_AGENT_MODE=1 env var (preferred — already set globally)
• Pass --agent flag explicitly if env var may not be set

Agent mode is auto-detected from env vars: FORCE_AGENT_MODE, GITHUB_COPILOT, CLAUDE_CODE, CURSOR_AGENT, etc.

In CCA/CI: API keys are required (set via repository secrets). In Codespaces: API keys preferred; if not set, pup-ensure-auth.sh will fall back to OAuth login.

Codespaces OAuth Troubleshooting

OAuth in Codespaces requires special handling because pup auth login starts a callback server on 127.0.0.1:8000, but the OAuth redirect from Datadog hits the user's local machine, not the Codespace. Codespaces port forwarding can proxy this, but only if:

1. The pup process stays alive — it must be run detached (via setsid or in a persistent background process), not inline in a script or ephemeral shell session.
2. DD_TOKEN_STORAGE=file must be set — there is no OS keychain in Codespaces.

pup-ensure-auth.sh handles both of these automatically. If OAuth still fails:

1. The auth URL is printed to the console — open it in a browser and authorize.
2. If the redirect fails with ERR_EMPTY_RESPONSE, copy the full URL from the browser address bar (it contains ?code=...&state=...) and replay it inside the Codespace: curl -s "<callback-url>".
3. As a last resort, set DD_API_KEY + DD_APP_KEY as Codespaces secrets to bypass OAuth.

⚠️ Timestamps: pup >=0.22 requires millisecond epoch timestamps for absolute --from/--to values. Use $(($(date +%s) * 1000)) in shell or int(dt.timestamp() * 1000) in Python.

⚠️ Large metric output: pup metrics search with by {host} grouping can return hundreds of KB of JSON. This triggers "Output too large" truncation in tool results, making the data unreadable. Always extract just the latest value using jq or python in the same pipeline:

# ✅ CORRECT — extract latest value per group in one command, small output
pup metrics search --query '...' --from "$FROM" --to "$TO" -o json 2>/dev/null \
  | python3 -c "
import json, sys
raw = sys.stdin.read()
lines = raw.strip().split('\n')
if lines[0].startswith('Warning:'): raw = '\n'.join(lines[1:])
data = json.loads(raw)
for series in data.get('data', {}).get('series', data.get('series', [])):
    scope = series.get('scope', series.get('tag_set', ['?']))
    points = series.get('pointlist', series.get('points', []))
    if points:
        last = points[-1]
        val = last[1] if isinstance(last, list) else last.get('value', '?')
        print(f'{scope}: {val}')
"

# ❌ WRONG — save full JSON to file then try to read/parse it across multiple tool calls
pup metrics search ... -o json > /tmp/huge_result.json  # 600KB+, will waste many turns parsing

⚠️ Stderr warning: pup outputs Warning: OS keychain not available, using file storage (~/.config/pup/) to stdout (not stderr) when API keys are used. This breaks JSON piping (pup ... | python3 -c "json.load(sys.stdin)"). Always use the file-based pattern below:

# ✅ CORRECT — redirect to file, strip warning, then parse
pup metrics search --query '...' --from "$FROM" --to "$TO" -o json 2>/dev/null > /tmp/pup_result.json
python3 -c "
import json
with open('/tmp/pup_result.json') as f:
    raw = f.read()
lines = raw.strip().split('\\n')
if lines[0].startswith('Warning:'):
    raw = '\\n'.join(lines[1:])
data = json.loads(raw)
# ... process data ...
"

# ❌ WRONG — pipe breaks on warning line
pup metrics search ... -o json | python3 -c "import json, sys; data = json.load(sys.stdin)"

⚠️ Bash ! expansion: Metric queries containing ! (negation filter like !environment:lab) will trigger bash history expansion inside double quotes. Always use single quotes for pup query arguments:

# ✅ CORRECT — single quotes prevent ! expansion
pup metrics search --query 'avg:metric{!environment:lab} by {pod_name}' ...

# ❌ WRONG — bash interprets !environment as history expansion
pup metrics search --query "avg:metric{!environment:lab} by {pod_name}" ...

Quick Reference

Task	Command
Metrics (v2 timeseries)	pup metrics query --query="..." --from="1h"
Metrics (v1 classic)	pup metrics search --query="..." --from="1h"
Metric discovery	pup metrics list --filter="service_name.*"
Monitors	pup monitors list --tags="service:foo"
Monitor search	pup monitors search --query="service:foo"
Dashboards	pup dashboards list / pup dashboards get <id>
Events (free-text v2)	pup events search --query="..." --from="1h" --to="now"
Monitor alerts	python3 $HOME/.pi/agent/skills/datadog/tools/discover-monitors.py --team X --from 60m (wraps pup events list --filter 'sources:monitor')
Deploys	python3 $HOME/.pi/agent/skills/datadog/tools/discover-deploys.py --service $SERVICE --from 4h
Incidents	pup incidents list / pup incidents get <id>
SLOs	pup slos list / pup slos get <id> / pup slos status <id>
APM services	pup apm services stats --env production --from ... --to ...
APM dependencies	pup apm dependencies list --env production --from ... --to ...
APM entities	pup apm entities list --from ... --to ...
Traces (span search)	pup traces search --query="service:foo" --from="1h"
Traces (aggregation)	pup traces aggregate --query="service:foo" --compute="count" --group-by="resource_name"
Infrastructure hosts	pup infrastructure hosts list
Error tracking	pup error-tracking issues search --query="..."

Evidence Links (Metric Explorer URLs)

Use dd-explorer-url.py to build clickable Datadog Metric Explorer links. See references/pup-commands.md for full usage.

• ⛔ NEVER hand-write Datadog URLs. Always use _datadog_metric_url() or build_metric_url(). If a URL lacks #N4Ig..., it is broken and must be regenerated.
• Use literal & between query parameters — NEVER HTML-encode as &
• Use dd-validate-urls.py to verify URLs before publishing

Detailed Command Reference

See references/pup-commands.md for expanded examples of every pup command (metrics, APM, traces, events, monitors, dashboards, SLOs, incidents).

Logs

⛔ Datadog Logs are NOT available. Our API token lacks log read permissions. Use Splunk or Kusto for log queries instead (see $HOME/.pi/agent/skills/splunk/SKILL.md or $HOME/.pi/agent/skills/kusto/SKILL.md).

Rules

1. Always baseline compare — query current window, then same time 7 days ago
2. Check both directions — upstream AND downstream via pup apm dependencies list
3. Use time windows — prefer absolute timestamps for incidents, not relative "1h"
4. Parse JSON output — pipe through jq for filtering: pup ... -o json | jq '.data[]'
5. Record findings after analysis — save a structured summary in the response/todo state; use set_finding only in harnesses that provide it
6. All output is JSON — use -o json (default) for machine parsing, -o table for readable summaries