In an incident the priority order is: restore service → understand cause → prevent recurrence. Don't invert it by debugging a live outage while users suffer.

Mitigate first. Reduce impact NOW: roll back the recent deploy, flip the feature flag off, fail over, scale up, or shed load. The fastest safe path to "users are OK again" beats the elegant fix.
Stabilize, then investigate. Once impact is contained, you have room to find the real cause without the clock running.
Use your signals. Logs, metrics, traces, recent changes (a deploy/config change right before the incident is the prime suspect). Form one hypothesis at a time (see systematic-debugging).
Communicate. State what's impacted, what you're doing, and the next update time. Silence is worse than bad news.
Fix the root cause with a regression test, and only then re-enable / re-deploy.
Blameless post-mortem. Write up the timeline, root cause, and concrete action items to prevent recurrence (better alerting, a guard, a test). Focus on the system and process, not who.

Red flag: forward-fixing a live outage instead of rolling back; declaring it resolved without confirming recovery in the metrics.

Más de este repositorio

mismo repositorio

react-doctor

New1Direction/korgex

Catch bad React in your changes — run react-doctor's deterministic scan and fix what it flags

2026-06-055

security-scan

New1Direction/korgex

Run a verifiable security scan (vulns, secrets, misconfig) over code you wrote or deps you added, before calling the work done

2026-06-035

browser-automation

New1Direction/korgex

Drive a real browser verifiably — navigate, snapshot, act BY INDEX, extract; every step provable

2026-06-025

verifiable-orchestration

New1Direction/korgex

Run multi-agent work as ONE provable causal DAG — parallel subagents, an Orchestrate graph, an immutable spec-seed

2026-06-025

building-an-mcp-server

New1Direction/korgex

Author or connect a Model Context Protocol server so an agent gains new tools

2026-06-015

ci-cd-pipelines

New1Direction/korgex

Build automated pipelines that gate merges and ship reliably

2026-06-015

name	incident-response
description	When production is broken — stop the bleeding first, diagnose second, learn third
version	1
trust	built-in

In an incident the priority order is: restore service → understand cause → prevent recurrence. Don't invert it by debugging a live outage while users suffer.

Mitigate first. Reduce impact NOW: roll back the recent deploy, flip the feature flag off, fail over, scale up, or shed load. The fastest safe path to "users are OK again" beats the elegant fix.
Stabilize, then investigate. Once impact is contained, you have room to find the real cause without the clock running.
Use your signals. Logs, metrics, traces, recent changes (a deploy/config change right before the incident is the prime suspect). Form one hypothesis at a time (see systematic-debugging).
Communicate. State what's impacted, what you're doing, and the next update time. Silence is worse than bad news.
Fix the root cause with a regression test, and only then re-enable / re-deploy.
Blameless post-mortem. Write up the timeline, root cause, and concrete action items to prevent recurrence (better alerting, a guard, a test). Focus on the system and process, not who.

Red flag: forward-fixing a live outage instead of rolling back; declaring it resolved without confirming recovery in the metrics.