datadog

# Datadog Operations Skill Comprehensive Datadog automation: query APM/logs/metrics/RUM/database, create monitors/dashboards/synthetics, manage incidents, trigger workflows, analyze costs and LLM usage. ## Quick Setup ### macOS ```bash ./setup.sh ``` ### Linux ```bash ./setup-linux.sh ``` ### Windows (PowerShell as Administrator) ```powershell .\setup-windows.ps1 ``` ## Platform-Specific Setup ### macOS **Install jq** (required for JSON processing): ```bash # Homebrew (recommended) brew install jq # MacPorts sudo port install jq # Direct binary (Apple Silicon) curl -L -o ~/bin/jq https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-macos-arm64 chmod +x ~/bin/jq # Direct binary (Intel) curl -L -o ~/bin/jq https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-macos-amd64 chmod +x ~/bin/jq ``` **Set environment variables** in `~/.zshrc`: ```bash export DD_API_KEY="your_api_key" export DD_APP_KEY="your_application_key" export DD_SITE="datadoghq.com" ``` ### Linux **Install dependencies:** ```bash # Debian/Ubuntu sudo apt-get install -y jq curl bc # Fedora/RHEL sudo dnf install -y jq curl bc # Arch sudo pacman -S jq curl bc ``` **Set environment variables** in `~/.bashrc`: ```bash export DD_API_KEY="your_api_key" export DD_APP_KEY="your_application_key" export DD_SITE="datadoghq.com" ``` ### Windows **Option 1: WSL (Recommended)** ```bash wsl --install # Then run Linux setup inside WSL ./setup-linux.sh ``` **Option 2: Git Bash** Install Git for Windows, then run `./setup.sh` in Git Bash. **Option 3: Native PowerShell** ```powershell # Install jq winget install jqlang.jq # Or: choco install jq # Set environment variables [Environment]::SetEnvironmentVariable("DD_API_KEY", "your_key", "User") [Environment]::SetEnvironmentVariable("DD_APP_KEY", "your_app_key", "User") ``` Use Python scripts directly: `python python\script_name.py` ## Required Permissions Get keys from Datadog → Organization Settings → API Keys / Application Keys **For workflow automation (`trigger-workflow`):** Enable "Actions API Access" on your app key: Datadog → Organization Settings → Application Keys → Click key → Enable "Actions API Access" ## Quick Reference ### Investigation Scripts | Script | Purpose | Example | |--------|---------|---------| | `query-apm.sh` | Find slow endpoints | `bash scripts/query-apm.sh --service my-service --duration 1h` | | `search-logs.sh` | Search logs for errors | `bash scripts/search-logs.sh --query "status:error" --duration 1h` | | `query-security-signals.sh` | Find security threats | `bash scripts/query-security-signals.sh --severity critical` | | `query-watchdog.sh` | Anomaly detection | `bash scripts/query-watchdog.sh --service my-service` | | `query-metrics.sh` | Fetch metrics data | `bash scripts/query-metrics.sh --metric "system.cpu.user"` | | `analyze-usage-cost.sh` | FinOps cost analysis | `bash scripts/analyze-usage-cost.sh --duration 30d` | | `analyze-llm.sh` | LLM observability | `bash scripts/analyze-llm.sh --service my-llm-app` | | `query-slos.sh` | SLO status | `bash scripts/query-slos.sh --service payment-api` | | `query-service-catalog.sh` | Service metadata | `bash scripts/query-service-catalog.sh list` | | `query-database.sh` | DB performance | `bash scripts/query-database.sh --host postgres-prod` | | `query-rum.sh` | Frontend performance | `bash scripts/query-rum.sh --application abc-123` | | `query-kubernetes.sh` | K8s workloads | `bash scripts/query-kubernetes.sh --cluster prod` | | `query-containers.sh` | Container metrics | `bash scripts/query-containers.sh --duration 1h` | | `query-network.sh` | Network monitoring | `bash scripts/query-network.sh --duration 1h` | ### Automation Scripts | Script | Purpose | Example | |--------|---------|---------| | `manage-monitors.sh` | Create/mute monitors | `bash scripts/manage-monitors.sh list` | | `create-dashboard.sh` | Generate dashboards | `bash scripts/create-dashboard.sh --service my-service` | | `trigger-workflow.sh` | Execute workflows | `bash scripts/trigger-workflow.sh list` | | `manage-incidents.sh` | Incident management | `bash scripts/manage-incidents.sh list` | | `manage-synthetics.sh` | Synthetic tests | `bash scripts/manage-synthetics.sh list` | | `manage-on-call.sh` | On-call scheduling | `bash scripts/manage-on-call.sh list` | | `manage-status-pages.sh` | Status pages | `bash scripts/manage-status-pages.sh list` | | `verify-setup.sh` | Validate config | `bash scripts/verify-setup.sh` | ## Workflows ### Investigate Production Issue ```bash bash scripts/query-watchdog.sh --service affected-service --duration 24h bash scripts/query-apm.sh --service affected-service --duration 1h bash scripts/search-logs.sh --service affected-service --status error --duration 1h ``` ### Check Cost & Usage ```bash bash scripts/analyze-usage-cost.sh --duration 30d --product all bash scripts/analyze-usage-cost.sh --duration 30d | jq '.recommendations[] | select(.priority == "high")' ``` ### Monitor LLM Application ```bash bash scripts/analyze-llm.sh --service my-genai-app --duration 24h ``` ## Output Format All scripts return structured JSON to stdout, status messages to stderr: ```bash bash scripts/query-apm.sh --service my-service 2>/dev/null | jq '.summary' ``` ## Python Alternative Python versions available in `python/` directory: ```bash python3 -m venv .venv && source .venv/bin/activate pip install -r python/requirements.txt python python/query_apm.py --service my-service --json ``` ## Go CLI Alternative Single-binary CLI available in `../dd-skill-test-go/`: ```bash # Native on all platforms - no dependencies datadog-cli apm --service my-service --duration 1h datadog-cli logs --query "status:error" datadog-cli monitors list datadog-cli health ``` See `../dd-skill-test-go/README.md` for installation. ## Notes - **Advanced Scripts:** Some scripts require Python. Run `./setup.sh` first. - **Windows:** Use WSL, Git Bash, or Python scripts directly. - **Go CLI:** Recommended for Windows - native binary, no dependencies.