Execute Cohere incident response procedures with triage, mitigation, and postmortem.
Use when responding to Cohere API outages, investigating errors,
or running post-incident reviews for Cohere integration failures.
Trigger with phrases like "cohere incident", "cohere outage",
"cohere down", "cohere on-call", "cohere emergency", "cohere broken".
Execute Cohere incident response procedures with triage, mitigation, and postmortem.
Use when responding to Cohere API outages, investigating errors,
or running post-incident reviews for Cohere integration failures.
Trigger with phrases like "cohere incident", "cohere outage",
"cohere down", "cohere on-call", "cohere emergency", "cohere broken".
allowed-tools
Read, Grep, Bash(kubectl:*), Bash(curl:*)
version
1.0.0
license
MIT
author
Jeremy Longshore <jeremy@intentsolutions.io>
tags
["saas","ai","nlp","cohere"]
compatible-with
claude-code
Cohere Incident Runbook
Overview
Rapid incident response procedures for Cohere API v2 outages. Covers triage, mitigation, communication, and postmortem for Chat, Embed, Rerank, and Classify endpoints.
Cohere API returning errors?
├─ YES: Is status.cohere.com showing incident?
│ ├─ YES → Cohere-side outage. Enable fallback. Monitor status page.
│ └─ NO → Check our API key and configuration.
│ ├─ 401 → API key revoked or wrong. Check CO_API_KEY.
│ ├─ 429 → Rate limited. Check if trial key in prod.
│ ├─ 400 → Bad request. Check request format (v1 vs v2?).
│ └─ 5xx → Cohere server issue. Retry with backoff.
└─ NO: Is our app healthy?
├─ YES → Intermittent. Monitor.
└─ NO → Our infrastructure. Check pods, memory, network.
Immediate Actions by Error Type
401 — Authentication Failure
# Verify API key is set
kubectl get secret cohere-secrets -o jsonpath='{.data.CO_API_KEY}' | base64 -d | head -c4
echo"..."# Test key directly
curl -s -o /dev/null -w "%{http_code}" \
-H "Authorization: Bearer $(kubectl get secret cohere-secrets -o jsonpath='{.data.CO_API_KEY}' | base64 -d)" \
-H "Content-Type: application/json" \
https://api.cohere.com/v2/chat \
-d '{"model":"command-r7b-12-2024","messages":[{"role":"user","content":"test"}]}'# If key is invalid: rotate# 1. Generate new key at dashboard.cohere.com# 2. Update secret
kubectl create secret generic cohere-secrets \
--from-literal=CO_API_KEY=NEW_KEY \
--dry-run=client -o yaml | kubectl apply -f -
# 3. Restart pods
kubectl rollout restart deployment/app
429 — Rate Limited
# Check if using trial key in production (trial = 20 calls/min)
KEY_LEN=$(kubectl get secret cohere-secrets -o jsonpath='{.data.CO_API_KEY}' | base64 -d | wc -c)
echo"Key length: $KEY_LEN chars"# If trial key: upgrade to production key at dashboard.cohere.com# If production key: reduce concurrency
kubectl setenv deployment/app COHERE_MAX_CONCURRENT=5
# Enable request queuing
kubectl setenv deployment/app COHERE_QUEUE_ENABLED=true
5xx — Cohere Server Errors
# Enable graceful degradation
kubectl setenv deployment/app COHERE_FALLBACK_ENABLED=true# If using RAG: fall back to rerank-only (skip chat)# If using agents: fall back to cached responses# Monitor Cohere status page for resolution
watch -n 30 'curl -s https://status.cohere.com/api/v2/status.json | jq .status.description'
P[1-4] INCIDENT: Cohere Integration
Status: INVESTIGATING / MITIGATED / RESOLVED
Impact: [e.g., "RAG answers unavailable, chat degraded to cached responses"]
Root cause: [e.g., "Cohere API returning 503 — confirmed on status.cohere.com"]
Current action: [e.g., "Enabled fallback mode, monitoring for recovery"]
Next update: [time]
External (Status Page)
Cohere Integration — Degraded Performance
Some AI-powered features may be slower than usual or temporarily unavailable.
We are monitoring the situation and will provide updates as available.
Last updated: [timestamp]
