inference-api-debugging

name

inference-api-debugging

description

Use when the inference API is paging — a p99 latency spike, a 429 surge, an elevated 5xx rate, a model returning gibberish, or a "the API is slow" Slack thread. Maps each symptom to the exact tools and query patterns that isolate the cause across metrics, logs, and traces, then emits a structured incident report. Reach for this the moment a dashboard or alert points at the highest-traffic inference API; do not hand-roll ad hoc queries.

Inference API Debugging

Overview

The inference API is the highest-traffic surface at the company: every external customer request and most internal traffic flows through one gateway into a fleet of model servers backed by GPUs. When it degrades, the blast radius is everyone.

This skill exists so you do not start from a blank page during an incident. It maps a symptom (the thing the alert or Slack thread is screaming about) to the tools that hold the answer and the exact query patterns to run against them, then funnels everything into a single structured report you can paste into the incident channel.

The investigation always follows the same spine:

1. Confirm the symptom — is it real, and is it API-wide or scoped to one model?
2. Walk the symptom→tool map in references/runbook.md — run the queries it prescribes, in order.
3. Separate cause from symptom — a 5xx spike is usually downstream of something (GPU OOM, a bad deploy, a saturated rate limiter).
4. Emit the structured report using the template at the end of the runbook.

When to Use

Use this skill when you see any of these symptoms on the inference API:

• p99 / p95 latency spike — the latency SLO alert fired, or customers report "the API is slow."
• 429 surge — a jump in rate-limit rejections, quota-exceeded errors, or a customer escalation about being throttled.
• Elevated 5xx rate — 500/502/503 above baseline, gateway error-rate alert.
• Model returning garbage — empty completions, truncated output, wrong model responding, or a quality regression report.
• Throughput drop — tokens/sec or requests/sec fell off a cliff while latency may look normal.

Do NOT use this skill when:

• The symptom is on a different service (billing UI, console, training cluster). This skill's queries target the inference gateway and model-server fleet only.
• You already have a confirmed root cause and just need to remediate — go straight to the relevant runbook/playbook for that fix.
• You are doing capacity planning or a non-incident investigation; this is a break-glass tool, not a reporting tool.

How to Use

1. Read references/runbook.md. It contains the full symptom→tool→query table. Find the row matching your symptom.

2. Run the prescribed queries in order. Each row gives you a metric query, a log query, and (where relevant) a trace lookup. Run them and record the numbers — you will need them for the report.

3. Always split aggregate vs per-model. The single most common mistake on this API is reading the aggregate dashboard and missing that one model is dragging the whole p99. Every latency/error query has a group by model variant in the runbook — use it.

4. Trace the worst offender. Once you have a slow or failing model, pull a representative trace (the runbook gives the trace query). The trace tells you which hop — gateway, queue, model server, tokenizer — owns the time or the error.

5. Classify cause vs symptom. Use the "common cause chains" section of the runbook. Example: elevated 5xx → trace shows model-server hop → model-server logs show CUDA OOM → root cause is GPU OOM surfacing as a generic 500, not a gateway bug.

6. Emit the structured report. Fill in the template at the bottom of references/runbook.md and post it to the incident channel.

Gotchas

• ALWAYS use p99 (or p95), never the average, for latency. Averages hide the tail. A mean latency of 400ms can sit on top of a p99 of 30s when one model or one GPU node is timing out. The SLO and the customer pain both live in the tail.
• ALWAYS split per-model before concluding. Aggregate metrics blend a healthy high-volume model with a sick low-volume one. The aggregate can look fine — or look uniformly bad — while the truth is one model. Run the group by model variant first, not last.
• GPU OOM shows up as a generic 5xx, not as "OOM." A CUDA out-of-memory on a model server gets caught and returned as a plain 500/503 at the gateway. If you see 5xx with no obvious gateway cause, go straight to model-server logs and grep for CUDA out of memory / OOM — do not assume the gateway is the problem.
• Label cardinality is capped — don't group by request_id or customer_id in the metrics system. High-cardinality labels get dropped or rejected by the metrics backend, so a group by request_id query silently returns nothing or partial data. Use metrics for low-cardinality dimensions (model, region, status_code) and pivot to logs/traces for anything per-request.
• A 429 surge is not always a problem with you. It can be a single customer hammering a key (check the per-key quota table) or an intentional limit doing its job. Confirm whether the rejections are concentrated on one key before paging the rate-limiter team.

Files

• SKILL.md — this file: when to trigger, the investigation spine, gotchas.
• references/runbook.md — the symptom→tool→query-pattern table, common cause chains, and the structured incident report template. Read this during an incident.