| name | trace-debugging |
| description | Use when investigating a specific trace_id — following a single request across services to understand where it failed, hung, or behaved unexpectedly. Triggers on "trace_id", "follow this request", "distributed trace", "trace across services", "what happened to request X". |
Trace Debugging
Use this skill when you have a trace_id and need to understand the end-to-end behavior of a single request.
When to dispatch the agent instead
If trace_request returns >200 logs OR the trace spans >5 services, stop and dispatch the graylog-trace-analyzer subagent. It returns a synthesized timeline (~50 lines) instead of flooding the main context with raw logs. Inline investigation is for smaller, focused traces.
Playbook
Step 1 — Pull the trace
Call trace_request with:
traceId: the trace ID the user gave you (or you extracted from a Sentry/dashboard URL).from / to: a generous window — ±15 minutes around the suspected event. Trace IDs are unique enough that overshooting is cheap.limit: 200 to start. Raise to 500 if you see truncation hints.
The response groups logs by service and orders by timestamp. Read it once before doing anything else.
Step 2 — Identify the failure span
Scan the trace for:
- The first
error or fatal level entry — that is the proximate failure.
- The originating service — usually the entry-point (gateway, API server) or the service that emitted the first error.
- Latency outliers — gaps > 1s between consecutive spans often indicate a hung dependency.
Step 3 — Pull surrounding context for each error span
For each error span, call get_surrounding_logs with:
centerTimestamp: the error timestamp.source or pod: filter to the same pod, since adjacent context from other pods is noise.secondsBefore / secondsAfter: start with 5 each.
The error message itself is rarely the cause. The cause is usually 1–10 lines before, in the same pod, often at info or warn level (e.g., a connection timeout, a config reload, a retry exhaustion).
Step 4 — Build the timeline
Assemble a human-readable summary:
- Originating service / pod
- Trace duration (first timestamp to last)
- Error spans in chronological order, each with: timestamp, service, message, likely cause from surrounding context
- Propagation: did the error originate downstream and bubble up, or originate at the entry point?
Pitfalls
- Window too narrow. Long-running jobs (batch processors, background workers) can have trace IDs alive for hours. If
trace_request returns very few results, widen the window before assuming the trace is small.
- Missing trace_id on some lines. Not every log emits trace_id. The trace_request results only show lines tagged with the ID. Use
get_surrounding_logs to pull adjacent untagged context.
- Re-used trace IDs. Rare but possible if a service is misconfigured to reuse IDs. If you see two clearly-different requests under one ID, widen
from/to and check the timestamps.
- Clock skew. Pods on different hosts can have skewed clocks. A "downstream first" timeline order can be an artifact of skew, not actual causation.
What to return to the user
A 5–10 line synthesis:
- Originating service / pod
- Root-cause line (from surrounding context, not the propagated error)
- Blast radius (how many downstream services were affected)
- Suggested next step (look at deploy times? check dependency health? read a specific service's recent commits?)
