| name | incident |
| description | Respond to production incidents on fancai.ru. Use when the site is down, slow, returning errors, or AI features are broken. |
| allowed-tools | Bash, Read, Grep, Glob |
Production Incident Response
Triage (< 2 minutes)
1. Is the site up?
curl -s -o /dev/null -w '%{http_code} %{time_total}s' https://fancai.ru
curl -s -o /dev/null -w '%{http_code} %{time_total}s' https://fancai.ru/api/v1/health
2. Are containers running?
ssh fancai "cd /opt/fancai/app && docker compose -f docker-compose.prod.yml ps"
3. Resource usage
ssh fancai "docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}'"
4. Recent logs (last 5 min)
ssh fancai "cd /opt/fancai/app && docker compose -f docker-compose.prod.yml logs --since 5m backend celery-worker"
Decision Tree
| Symptom | Likely Cause | Action |
|---|
| 502/504 | Backend crashed | Restart backend |
| Slow (>3s) | DB/memory | Check PG connections, memory |
| Images broken | OpenRouter down | /openrouter-monitor |
| Books stuck | Celery OOM | /celery-debug |
| Entities missing | Extraction failed | /entity-pipeline |
| SSL error | Caddy cert | Check Caddy logs |
Recovery Commands
ssh fancai "cd /opt/fancai/app && docker compose -f docker-compose.prod.yml restart backend"
ssh fancai "cd /opt/fancai/app && docker compose -f docker-compose.prod.yml down && docker compose -f docker-compose.prod.yml up -d"
Post-Incident
- Document in
docs/reports/YYYY-MM-DD-incident-<desc>.md
- Identify root cause
- Update monitoring if blind spot found
