log-correlator

name

log-correlator

description

Use when you have one request ID and need its whole story — a customer escalation citing a request_id, a trace that dead-ends, or a "what happened to this specific request" question during an incident. Fans out to every system that touched the request (gateway, model server, billing, traces), merges by request_id, and prints one timeline sorted by timestamp. Reach for this instead of grepping each log store by hand.

Log Correlator

Overview

A single inference request touches several systems on its way through: the gateway accepts it, a model server runs it, billing meters it, and a trace records the spans. When something goes wrong with one specific request, the answer is spread across all of those stores, each with its own UI, query language, and clock. Stitching them together by hand is slow and error-prone — exactly the wrong thing to be doing while a customer waits.

scripts/correlate.py takes a request ID, fans out to every system in parallel, merges the results by request_id, sorts the merged events by timestamp, and prints one unified timeline. You get the request's whole story in one view.

When to Use

Use this skill when:

• A customer escalation cites a request_id — "request req_a1b2c3 failed, what happened?"
• A trace dead-ends — you have a request_id from a partial trace and need the log lines the trace didn't capture (sampling dropped spans, or a hop wasn't instrumented).
• You're root-causing one request during an incident and need to see the gateway → model-server → billing handoffs in order.

Do NOT use this skill when:

• You don't have a request_id — for symptom-first or aggregate investigation, use inference-api-debugging; for alert triage, use oncall-runner.
• You need aggregate behavior across many requests — this tool is single-request by design.

How to Use

Run the script with the request ID:

python scripts/correlate.py req_a1b2c3d4

The script will:

1. Fan out to all four systems (gateway logs, model-server logs, billing, traces) concurrently, querying each for the request_id.
2. Merge the returned events into one list, tagging each with its source system.
3. Sort by timestamp (normalized to UTC) and print a single timeline, one line per event: timestamp source level message.
4. Flag correlation problems at the bottom — systems that returned nothing, suspected clock skew, and a note if the trace looks sampled-out.

Options:

• --since 1h / --window 2h — bound the query window (helps when retention differs across systems).
• --systems gateway,model-server — restrict the fan-out to specific systems.
• --json — emit the merged timeline as JSON for further processing.

Read the timeline top to bottom: the last event before things go quiet, or the first error-level line, is usually where the request died.

Gotchas

• request_id propagation has gaps — absence in a system is a clue, not proof of nothing. A downstream hop may receive the request under a different correlation field (e.g. billing keys off meter_id carrying the request_id in a sub-field), or may simply not log the id at all. If a system returns zero events, check whether the id is propagated there before concluding the request never reached it. The script flags empty systems explicitly for this reason.
• Clocks are skewed between systems — sort carefully and don't over-read sub- second ordering. Gateway, model-server, and billing run on different hosts with independent clocks; skew of tens to hundreds of milliseconds is normal. The script normalizes to UTC and sorts, but two events milliseconds apart from different systems may actually be in the opposite order. Trust causal ordering (a response can't precede its request) over raw timestamp ordering when they conflict; the script warns when it detects a likely-skewed adjacent pair.
• Retention windows differ — an old request may be gone from some stores but not others. Model-server debug logs may retain for days while gateway access logs retain for weeks. A partial timeline for an old request_id is often a retention artifact, not a sign the request skipped a hop. Use --since/--window to set expectations, and read the per-system retention note the script prints.
• Traces are sampled — a missing trace doesn't mean the request didn't run. Distributed tracing samples a fraction of requests, so many perfectly normal requests have no trace at all. If the trace system returns nothing but the logs show a full successful path, that's sampling, not a problem. The script notes when traces are empty but logs are present.

Files

• SKILL.md — this file: when to trigger, how to run the script, gotchas.
• scripts/correlate.py — fans out to gateway logs, model-server logs, billing, and traces for a request_id; merges by request_id, sorts by timestamp, prints a unified timeline; flags empty systems, clock skew, and sampled-out traces.