production-incident

name	production-incident
description	Use when investigating a production incident, outage, error spike, or unexpected behavior in production services. Also use when the user says "production issue," "site is down," "errors in prod," "something broke," or describes urgent production symptoms.

Production Incident Response

Overview

Evidence-first incident diagnosis. Run parallel diagnostics, build a structured report, and get approval before touching anything.

Core rule: Do NOT edit any files or suggest fixes until the diagnostic report is presented and the user approves a plan.

When to Use

• Production outage or degraded service
• Error spikes in logs or monitoring
• Unexpected behavior reported by users
• Performance degradation
• Failed deployments causing issues

Diagnostic Process

Step 1: Gather Context

Ask the user:

• What symptoms are they seeing? (errors, slowness, downtime)
• When did it start?
• Any recent deployments or changes?
• Which service/app is affected?

Step 2: Run Parallel Diagnostics

Spawn all 5 diagnostic agents in parallel using the Agent tool:

Agent	Task	How
DB Schema	Read full schema	Read prisma/schema.prisma or equivalent migration files. Never guess column/table names.
PM2 Logs	Last 30 min of logs	pm2 logs --lines 500 --nostream via Bash. Look for errors, warnings, restarts.
Grafana Metrics	Error rates, CPU, memory	Query Grafana MCP for error counts, error rate trends, resource utilisation.
Environment	Understand config chain	Read ecosystem.config.js and .env files. Map how env vars are loaded.
Recent Changes	Last 5 commits	git log -5 --oneline --stat via Bash. Identify what changed recently.

Adapt these to the project:

• No PM2? Check Docker logs (docker logs --since 30m <container>).
• No Prisma? Check migration files, Eloquent models, or raw schema.
• No Grafana MCP? Ask user for access or skip with note.

Step 3: Present Incident Report

Structure the report exactly as follows:

## Incident Report

### Timeline
- [When symptoms started]
- [When first reported]
- [Key events from logs]

### Root Cause Hypothesis
[What the evidence points to, with specific log lines/metrics as proof]

### Evidence
- **Logs:** [relevant error messages]
- **Metrics:** [error rates, resource usage]
- **Recent changes:** [commits that may be related]
- **Config:** [any env/config issues found]

### Affected Services
- [List of services impacted]

### Proposed Fix
[Specific changes to make]

### Rollback Plan
[How to revert if the fix makes things worse]

Step 4: Wait for Approval

Do NOT proceed with any fix until the user explicitly approves. Present the report and wait.

Red Flags - Things That Make Incidents Worse

• Guessing at DB column names instead of reading the schema
• Modifying DATABASE_URL or env vars without understanding the loading chain
• Restarting services before understanding the root cause
• Reading stale logs and presenting them as current
• Exploring local codebase when the issue is on a remote server

Common Mistakes

Mistake	Prevention
Guessing schema	Always read prisma/schema.prisma first
Wrong env chain	Read .env AND ecosystem.config.js before touching env vars
Stale log data	Use --since flags, check timestamps
Fixing symptoms not cause	Trace the full error path before proposing fixes
Skipping rollback plan	Every fix needs a way to undo it