groq-incident-runbook

name	groq-incident-runbook
description	Execute Groq incident response: triage, mitigation, fallback, and postmortem. Use when responding to Groq-related outages, investigating errors, or running post-incident reviews for Groq integration failures. Trigger with phrases like "groq incident", "groq outage", "groq down", "groq on-call", "groq emergency", "groq broken".
allowed-tools	Read, Bash(kubectl:*), Bash(curl:*)
version	1.11.0
license	MIT
author	Jeremy Longshore <[e-mail]>
tags	["saas","groq","incident-response"]
compatibility	Designed for Claude Code, also compatible with Codex and OpenClaw

Groq Incident Runbook

Overview

Rapid incident response procedures for Groq API failures. Groq is a third-party inference provider -- when it goes down, your mitigation options are: wait, fall back to a different model, or fall back to a different provider.

This SKILL.md is the high-level flow. Deep, copy-paste-ready material lives in references/:

• triage-and-diagnostics.md — severity table, the Quick Triage script, and the full decision tree.
• mitigations.md — fallback-model routing (TypeScript), 429 rate-limit actions, 401 key rotation.
• communication-and-postmortem.md — Slack/status-page templates, evidence collection, postmortem template.

Prerequisites

• GROQ_API_KEY exported in the environment you run the triage commands from.
• curl for API probes; kubectl only if you collect logs from a Kubernetes deployment.
• Access to console.groq.com to rotate keys or upgrade the plan.
• A configured fallback provider (e.g. OpenAI) if you need to fail away from Groq entirely.

Authentication: every Groq API call in this runbook authenticates with a bearer token — Authorization: Bearer $GROQ_API_KEY. Keep the key in a secret manager, never inline; the evidence-collection step in communication-and-postmortem.md redacts gsk_ tokens from logs before archiving.

Instructions

Work the incident in five phases. Each phase points to the reference file with the exact commands.

1. Classify severity. Match user impact to the P1–P4 table in triage-and-diagnostics.md — this sets your response-time budget (P1 < 15 min, P4 next business day).

2. Triage. Run the Quick Triage script (status reachability, auth, per-model availability, rate-limit headers). The one-line probe that starts most incidents:

   curl -s -o /dev/null -w "%{http_code}\n" \
     https://api.groq.com/openai/v1/models \
     -H "Authorization: Bearer $GROQ_API_KEY"
   

3. Decide. Walk the decision tree in triage-and-diagnostics.md to turn the HTTP code (timeout / 401 / 429 / 5xx / slow) into an action path.