fix-datadog-issues

# Fix Datadog Error Tracking issues This skill takes a brand-new agent from "look at Datadog" all the way to a reviewable PR. You will run it repeatedly across **separate, context-free sessions**, so it is written to be restartable: a fixed error simply stops receiving occurrences, and (when the tooling allows) you leave a comment on the issue so the next agent does not redo your analysis. The Datadog MCP server is `plugin:datadog:mcp`. Its tools are **deferred** — their schemas load on demand. Load them with `ToolSearch` (e.g. `select:mcp__plugin_datadog_mcp__aggregate_spans`) before calling, and run the server's skill-discovery first (see §1). --- ## The pipeline (overview) ``` 0. Setup & disambiguate → 1. Navigate Datadog → 2. Pick an issue → 3. Root-cause in code → 4. ⛔ CONFIRM PLAN WITH USER → 5. Reproduce with tests + fix → 6. Comment on the issue → 7. PR to development ``` **Hard gate at step 4:** you investigate freely, but you **do not** write a fix, create a branch, or open a PR until the user has seen your plan and approved it (see §4). The only exception is explicit pre-authorization (e.g. "just fix it and PR"). Do **not** try to fix every issue. Scope a PR by size/risk (see §2.4): one big/critical bug = its own PR; a few small, independent, related bugs may share one PR (cap ~4). --- ## 0. Setup and critical disambiguations Read these first — each one was a real wall that cost time. - **"Incidents" almost always means Error Tracking *issues*, not Incident Management.** Datadog Incident Management (`search_datadog_incidents`) is typically **empty (0)** here, and incidents have no "occurrences". When the user says incidents / issues / errors / "most occurrences" / "newest", they mean **Error Tracking** (errors grouped into issues, with occurrence counts and trends). - **Only v2 services are fixable from this repo.** This repo (`latitude-v2`, trunk `development`) owns: `api`, `ingest`, `web`, `workers`, `workflows`. The **`latitude-llm-*`** services (`latitude-llm-web`, `latitude-llm-workers`, `latitude-llm-gateway`, …) are the **legacy v1** codebase (branch `latitude-v1`) — **out of scope** unless the user says otherwise. Derive the live v2 list from `ls apps/` so it never goes stale. - **Production only.** Filter `env:production`. Error Tracking buckets by `env`; staging issues exist separately (`env:staging`) but are out of scope unless the user hands you a specific staging issue. - **The Latitude MCP server (`mcp__latitude__*`) is NOT this.** `mcp__latitude__listIssues` / `getIssue` / `resolveIssues` operate on issues detected in *Latitude customers' LLM traces* — a different product surface. Never use them to triage our own app's runtime errors. Use `plugin:datadog:mcp`. - **Error Tracking toolset must be enabled.** This skill needs the `error-tracking` toolset (stable, not on by default). If `ToolSearch` for `datadog error tracking issue` finds no `mcp__plugin_datadog_mcp__*error_tracking*` tools, the toolset is off — tell the user to enable it via the `/datadog:ddtoolsets` skill (add `error-tracking`), then `/reload-plugins` + re-auth. You can still do everything *except* read/write issue objects directly via the **span fallback** in §1.3. --- ## 1. Navigating Datadog ### 1.1 Always start with skill discovery The server ships domain guides that are not visible in tool names. In parallel: - `load_datadog_skill(skill_name="datadog/traces")` — span query syntax & attributes. - `list_datadog_skills(query="error tracking ...")` — find the right guide. Load `datadog/logs` if you pivot to logs. Skip re-loading a guide you already loaded this session. ### 1.2 The lay of the land - **Org/site:** `datadoghq.eu` (UI: `app.datadoghq.eu`, MCP domain `mcp.datadoghq.eu`). - **Services:** `search_datadog_services` lists everything (apps, DB adapters like `*-postgres`/`*-aws-s3`, and external hosts like `api.openai.com`). Only the bare v2 app names matter here. - **`service.version` == git commit SHA.** Every span tags the deployed SHA. This is gold: it maps an error to a commit and lets you correlate with deploys via `get_change_stories`. ### 1.3 Where Error Tracking issues live, and how to read them An **issue** is a fingerprinted group of error occurrences. Two ways in — use both: **(a) Error Tracking tools (preferred; needs the toolset).** Discover exact names at runtime: `ToolSearch(query="datadog error tracking issue")`. Expect at least `get_datadog_error_tracking_issue` (by issue id) and a list/search tool; there may be an **update/state** and/or **comment** tool — confirm their real names/schemas before relying on them (see §6). **(b) Span aggregation (always works, even with the toolset off).** Issues are stamped onto error spans: - `custom.issue.id` — the issue UUID (fetch with `custom_attributes:["issue.*"]`). - `issue.first_seen` (epoch ms), `issue.first_seen_version` (git SHA of first occurrence), `issue.age`. - Plus `@error.type`, `@error.message`, `error.stack`, `resource_name`, `service`, `env`. **Find the heavy hitters** (the workhorse query): ``` aggregate_spans( query = "status:error env:production", from = "now-7d", to = "now", computes= [{field:"*", aggregation:"COUNT", output:"count", sort:"desc"}], group_by= {fields:["service","@error.type"], limit:40} ) ``` Then narrow into messages/resources for the candidates you care about: ``` aggregate_spans(query="service:workers status:error env:production @error.type:(TypeError OR RepositoryError)", group_by={fields:["@error.message","resource_name"], limit:25}, computes=[COUNT desc]) ``` **Read raw detail** (stack, http, issue id) for a specific group: ``` search_datadog_spans( query = "service:web status:error env:production @error.type:Error resource_name:GET", custom_attributes = ["error.*","http.*","issue.*"], max_tokens = 7000) ``` **Link to an issue** (needed in the §4 report and the §7 PR). Prefer the canonical URL the error-tracking tool returns. Otherwise build it from the org base + issue id and open it to confirm it resolves: `https://app.datadoghq.eu/error-tracking/issues/<issue.id>`. (Span search responses also return a `base_url` and a `traces_explorer_url` you can fall back to.) ### 1.4 Query pitfalls (these bit us) - **`@error.message` is not reliably wildcard/full-text searchable.** `@error.message:"foo*"` may return 0. Instead **group by `@error.message`** in `aggregate_spans`, or filter by `@error.type` + `resource_name` and read messages from raw spans. - Reserved attrs take **no `@`**: `service`, `resource_name`, `status`, `type`, `trace_id`. Span attrs take `@`: `@error.type`, `@http.status_code`, `@duration` (**nanoseconds!**). - Group multi-values: `@error.type:(A OR B)`, not `@error.type:A OR @error.type:B`. --- ## 2. Pick an issue ### 2.1 If the user named an issue Target it directly (by issue id/slug/url, or a quoted error message → resolve via the queries above). Skip the ranking; go to §3. ### 2.2 Otherwise: the funnel (classification *before* ranking) **Occurrence count alone is a trap.** Most high-count "errors" are not fixable code bugs. Filter first. **Stage A — scope gate (cheap, 1–2 aggregate calls):** v2 service, `env:production`, `status:error`, `now-7d`. Inspect the **top ~15** by count. Ignore issues with **< 5 occurrences** in the window unless they are new+rising or user-specified (sub-5 are usually non-reproducible one-offs). **Stage B — classify each candidate** (read its message + stack). Bucket it: | Class | Signatures (examples) | Default action | |---|---|---| | **A. Genuine code bug** | `TypeError`, null/undefined deref, validation/logic errors, data-handling (bad UTF-8 / lone surrogates, encoding, parsing) | **Candidate to fix** | | **B. Infra / transient** | `Timeout`, `socket hang up`, `ECONNRESET`, `deadlock detected`, `timeout exceeded when trying to connect`, 429, 503, pool exhaustion | Usually **not** a code fix. Note & skip (resilience/retry work only if asked) | | **C. Deploy / version skew** | "Server function info not found for `<hash>`", "Failed to fetch dynamically imported module" (old `first_seen_version` ≠ current `service.version`; stale client tabs) | Framework-level graceful handling, **not** a logic bug | | **D. Expected / not-an-error** | BullMQ `DelayedError`, `*LockUnavailableError` that is retried with backoff, a `NotFoundError` that callers handle | **Noise** — ignore | | **E. Upstream / third-party** | external provider 5xx, provider rate limits with correct handling | Not ours | Only **Class A** proceeds. For B–E: leave a one-line verdict comment on the issue if you can (§6), then skip. **Stage C — rank the Class-A bugs** on three axes, then pick the top: 1. **Impact** — customer-facing (`web`/`api`) > background (`workers`/`workflows`); data corruption/loss > transient failure; *silent-wrong* > *loud-fail*; does it block a user flow? 2. **Volume × trend** — occurrences **and** direction. Use the 14-day trend; rising/new beats flat/decaying. 3. **Fix confidence × blast radius** — clear, bounded root cause + small change = high ROI; sprawling or unknown = defer. Pick = highest **(impact × trend)** among the **confidently fixable**. ### 2.3 The recency premium (catch regressions early) Give extra weight to **new + rising** issues even at lower absolute count: - "New" = `issue.first_seen` within **~72h**, OR `first_seen_version` is one of the **last 1–2 deploys**. - A recent first-seen usually means a fresh regression — `first_seen_version` + `get_change_stories` often hand you the **culprit commit**, making the fix faster and higher-confidence, and catching it early prevents pile-up. An ancient, flat, high-count issue is a yellow flag, not an automatic top pick. ### 2.4 How many to fix / how to batch - **Solo PR** if: critical or large; touches core/shared code; or the root cause is non-trivial. - **Group 2–4** into one PR only if **all** are: small, independent, low-risk, **and** thematically related (same subsystem → one reviewer context). Never mix a risky fix with trivial ones. Cap ~4. --- ## 3. Root-cause in code Use the `analyze-problem` skill's method. Then, specific to this workflow: - **Generalize past the observed symptom.** One issue is often one *instance* of a broader bug. Example: a Voyage embeddings `400 invalid UTF-8` and a ClickHouse `missing second part of surrogate pair` were **one** root cause — unsanitized lone UTF-16 surrogates hitting two sinks. Fix the **source**, not each sink, and look for **sibling call sites** with the same flaw. - **Map the error to code from the span:** `service` → `apps/<svc>`; `resource_name` (e.g. bullmq `process <queue>`, or `GET /…`) → the handler/job; `error.stack` frames → the throwing module. Confirm the deployed SHA (`service.version`) matches what you're reading. - **Find the commit that introduced it.** `issue.first_seen_version` is the git SHA of the first occurrence — the deploy that introduced the regression. Once you've located the faulty line(s), run `git blame`/`git log -S '<symbol>' -- <file>` to name the culprit commit, and `get_change_stories` to see that deploy in context. Capture both the `file:line` and the commit SHA — §4 asks for them. - **Decide fixability honestly.** If it's Class B/C/D/E in disguise, or the fix needs product/infra decisions beyond code, say so and record it (§6) instead of forcing a fake fix. - Respect the architecture (`architecture-boundaries`): fix at the right layer (domain use-case vs platform adapter vs app boundary). Prefer the layer the codebase already uses for that concern — search for an existing helper before writing a new one. --- ## 4. Checkpoint — confirm the plan with the user (do not skip) **Stop here. Do not write a fix, create a branch, or open a PR until the user approves.** This is a hard gate: everything up to now is read-only investigation. Report back with a concise plan and wait. Present it in this order — be organized, not a wall of text: 1. **Issues found** — a short list (table is fine) of the candidates from §2. For each, give: the error signature, the service, occurrences + 14-day trend, the class (§2.2), a **link to the Datadog issue** (§1.3), and a **one-line description of what it actually is** — not just the raw message. Make clear which ones you ruled out and why. 2. **What you're focusing on** — the issue(s) you chose and **why** (impact × trend × confidence), plus what you deliberately skipped with the class reason (infra / version-skew / not-an-error / upstream). 3. **Hypothesis** — for each chosen issue, the root cause in plain language, backed by concrete evidence: - **`file:line` references** to the code at fault (and sibling call sites if it generalizes); - the **commit that introduced it** where you can find it (`issue.first_seen_version` + `git blame` / `git log -S`, §3) — link it as `<repo>/commit/<sha>`; - whether this issue is one instance of a broader bug. 4. **Proposed fix** — what you'll change, at which layer, the blast radius, and the **test plan** (the reproduction plus the novel cases you'll add). 5. **PR plan** — single PR vs grouped (per §2.4), and the base branch (`development`). 6. **Assistance needed / blockers** — anything you need from the user, asked explicitly: ambiguous intent, a product/infra decision, missing access (e.g. the error-tracking toolset is off), an issue that looks **unfixable in code**, or a fix that turned out larger/riskier than expected. Then **wait for confirmation** and adjust the plan to their feedback before proceeding to §5. **Pre-authorization escape hatch:** if the user already said to fix and open the PR without checking back (e.g. "just fix it and PR", or a non-interactive/scheduled run with standing approval), state the plan briefly and proceed — but still **stop and ask** if you hit a blocker, an ambiguous choice, or a fix materially larger or riskier than what you described. --- ## 5. Reproduce with tests, then fix **Tests first, and broader than the single failure.** 1. Write a failing test that reproduces the bug **and** new, *different* inputs that exercise the same root cause (not just the one occurrence you saw). This both pins the bug precisely and guards the general case. Follow the `testing` skill (Vitest layers, PGlite/chdb testkit, `/testing` exports;