incident-management

name	incident-management
description	Implement incident management processes and escalation procedures. Configure on-call schedules and post-incident reviews. Use when managing production incidents.
license	MIT
metadata	{"author":"devops-skills","version":"1.0"}

Incident Management

Implement effective incident management processes including severity definitions, escalation matrices, war room procedures, and blameless post-mortem templates.

When to Use

• Establishing incident management processes for production systems
• Defining severity levels and escalation procedures
• Running war rooms and coordinating incident response
• Conducting blameless post-incident reviews
• Building on-call schedules and notification workflows
• Meeting compliance requirements for incident response (SOC 2, HIPAA, PCI DSS)

Severity Levels

severity_definitions:
  SEV1_critical:
    impact: "Complete service outage or data breach affecting all/most customers"
    examples:
      - Production site completely down
      - Data breach confirmed or suspected
      - Complete loss of a critical business function
      - Security incident with active exploitation
    response_time: "Immediate (within 5 minutes)"
    update_frequency: "Every 15-30 minutes"
    who_is_paged: "On-call engineer, engineering manager, incident commander, executive on-call"
    communication: "Status page update, customer email, executive notification"
    resolution_target: "< 1 hour to mitigate"

  SEV2_major:
    impact: "Major feature broken or severe degradation affecting many customers"
    examples:
      - Key feature completely non-functional
      - Significant performance degradation (>5x latency)
      - Data processing pipeline completely stalled
      - Partial outage affecting a region or segment
    response_time: "Within 15 minutes"
    update_frequency: "Every 30-60 minutes"
    who_is_paged: "On-call engineer, engineering manager"
    communication: "Status page update if customer-facing"
    resolution_target: "< 4 hours to mitigate"

  SEV3_moderate:
    impact: "Minor feature impaired or degradation affecting some customers"
    examples:
      - Non-critical feature broken
      - Moderate performance degradation
      - Elevated error rate (below threshold for SEV2)
      - Single-customer impact on non-critical function
    response_time: "Within 1 hour during business hours"
    update_frequency: "Every 2-4 hours"
    who_is_paged: "On-call engineer"
    communication: "Internal only unless customer inquires"
    resolution_target: "< 1 business day"

  SEV4_low:
    impact: "Cosmetic issue, minor inconvenience, or non-customer-facing problem"
    examples:
      - UI cosmetic bug
      - Non-critical monitoring gap
      - Internal tool degradation
      - Documentation inaccuracy in production
    response_time: "Next business day"
    update_frequency: "As needed"
    who_is_paged: "None (ticket created)"
    communication: "None"
    resolution_target: "Within sprint planning cycle"

Escalation Matrix

escalation_matrix:
  tier_1_on_call_engineer:
    reached_via: "PagerDuty / OpsGenie alert"
    responsibilities:
      - Acknowledge alert within 5 minutes
      - Assess severity and impact
      - Begin troubleshooting
      - Escalate to Tier 2 if unable to resolve within 30 minutes (SEV1/2)
    escalation_trigger: "Cannot resolve, needs additional expertise, or severity upgrade"

  tier_2_team_lead_or_sme:
    reached_via: "PagerDuty escalation or direct page"
    responsibilities:
      - Provide subject matter expertise
      - Assist with diagnosis and resolution
      - Coordinate with other teams if  
              
     

  
     
    
            
         
             
         
     

  
     
    
         
         
         
           
     

  
    
       
       
       
       
    
       
       
       

War Room Procedures

war_room:
  activation: "Automatically for SEV1, on-demand for SEV2"

  setup:
    communication_channel:
      primary: "Dedicated Slack channel (#incident-YYYY-MM-DD-brief-name)"
      voice: "Zoom/Google Meet bridge (persistent link)"
      backup: "Phone conference bridge"
    channel_rules:
      - "Only incident-related communication in the channel"
      - "Use threads for side discussions"
      - "Prefix messages with role (IC:, COMMS:, ENG:)"

  roles:
    incident_commander:
      responsibilities:
        - Own the incident from declaration to resolution
        - Coordinate all response activities
        - Make decisions on response actions
        - Assign tasks to responders
        - Determine when incident is resolved
        -  
       

    
      
              
           
            
           
            
       

    
      
             
           
             
           
       

    
      
              
           
              
             
       

  
    
       
       
       
       

    
       
       
       

    
       
       
       
       

    
       
       
       
       

    
       
       
       

On-Call Configuration

on_call_schedule:
  rotation_structure:
    primary:
      rotation: "Weekly"
      handoff: "Monday 10:00 AM local time"
      team_size: "Minimum 5 engineers in rotation"
    secondary:
      rotation: "Weekly (offset from primary)"
      activation: "If primary does not acknowledge within 10 minutes"

  expectations:
    response_time: "Acknowledge alert within 5 minutes"
    availability: "Reachable by phone and laptop within 15 minutes"
    handoff: "Document any ongoing issues during handoff"
    compensation: "Per company on-call compensation policy"

  health:
    max_consecutive_weeks: 2
    minimum_gap_between_rotations: "2 weeks"
    post_incident_rest: "If engaged for 4+ hours overnight, late start next day"
    burnout_monitoring: "Track pages per person per week, rebalance if needed"

  pagerduty_configuration:
    escalation_policy:
      - level_1:
          target: "Primary on-call"
          timeout: "5 minutes"
      - level_2:
          target: "Secondary on-call"
          timeout: "10 minutes"
      - 
           
           

    
      
         
         
         
      
         
         

Post-Mortem Template

# Post-Incident Review: [Incident Title]

**Date:** YYYY-MM-DD
**Severity:** SEV[1-4]
**Duration:** [Start time] to [End time] ([X hours Y minutes])
**Incident Commander:** [Name]
**Author:** [Name]
**Status:** Draft / In Review / Final

## Executive Summary
[2-3 sentence summary of what happened, the impact, and the resolution]

## Impact
- **Customer impact:** [Number/percentage of customers affected, what they experienced]
- **Duration of impact:** [How long customers were affected]
- **Revenue impact:** [Estimated financial impact, if applicable]
- **Data impact:** [Any data loss or corruption]
- **SLA impact:** [Any SLA breaches]

## Timeline (all times UTC)
| Time | Event |
|------|-------|
| HH:MM | [First anomaly detected by monitoring] |
| HH:MM | [Alert fired / customer report received] |
| HH:MM | [On-call engineer acknowledged] |
| HH:MM | [Incident declared at SEV level] |
| HH:MM | [War room established] |
| HH:MM | [Root cause identified] |
| HH:MM | [Mitigation applied] |
| HH:MM | [Service restored] |
| HH:MM | [Incident resolved] |

## Root Cause
[Detailed technical explanation of what caused the incident]

## Detection
- **How was the incident detected?** [Monitoring alert / customer report / manual observation]
- **Time to detect:** [Time from first anomaly to detection]
- **Could we have detected sooner?** [Yes/No, with explanation]

## Response
- **What went well:**
  - [List things that worked effectively during response]
  - [E.g., "Runbook for database failover was accurate and followed successfully"]
  - [E.g., "Communication to customers was timely and clear"]

 
 [List things that slowed or hindered response]
 [E.g., "Took 20 minutes to identify the correct service owner"]
 [E.g., "Monitoring did not alert on the specific failure mode"]


[List all factors that contributed to the incident occurring or being worse than it could have been. This is not about blame - it is about understanding the system.]

 [Factor 1: e.g., "Configuration change was not tested in staging"]
 [Factor 2: e.g., "Alert threshold was too high to catch gradual degradation"]
 [Factor 3: e.g., "No circuit breaker between Service A and Service B"]


| ID | Action | Owner | Priority | Due Date | Status |
|----|--------|-------|----------|----------|--------|
| 1 | [Preventive action] | [Name] | P1 | YYYY-MM-DD | Open |
| 2 | [Detection improvement] | [Name] | P2 | YYYY-MM-DD | Open |
| 3 | [Process improvement] | [Name] | P2 | YYYY-MM-DD | Open |
| 4 | [Runbook update] | [Name] | P3 | YYYY-MM-DD | Open |


[Key takeaways that should be shared broadly]


 [Link to monitoring dashboards during incident]
 [Link to relevant log queries]
 [Link to war room channel archive]

Post-Mortem Process

post_mortem_process:
  scheduling:
    sev1: "Within 48 hours of resolution"
    sev2: "Within 1 week of resolution"
    sev3: "Within 2 weeks (optional, based on learning potential)"
    sev4: "Not required"

  meeting_format:
    duration: "60-90 minutes"
    attendees:
      required: "IC, technical lead, scribe, involved engineers"
      optional: "Engineering manager, product manager, affected team leads"
    agenda:
      - "5 min: Review timeline and facts"
      - "15 min: Walk through root cause and contributing factors"
      - "15 min: Discuss what went well"
      - "15 min: Discuss what could be improved"
      - "15 min: Define and assign action items"
      - "5 min: Identify lessons learned and sharing plan"

  principles:
    - "Blameless: Focus on systems and processes, not individuals"
    - "Factual: Base discussion on data, logs, and observations"
    - "Forward-looking: Prioritize preventive actions over assigning fault"
    - "Complete: Address detection, response, and prevention"
    - "Actionable: Every finding should produce a tracked action item"

  action_item_tracking:
    - 
     
     
     
     
     

Incident Metrics

incident_metrics:
  mttr:
    name: "Mean Time to Resolve"
    definition: "Average time from incident detection to resolution"
    target: "SEV1: <1h, SEV2: <4h"
    trending: "Track monthly, aim for improvement"

  mttd:
    name: "Mean Time to Detect"
    definition: "Average time from incident start to detection"
    target: "< 5 minutes for SEV1/2"
    trending: "Monitors effectiveness of alerting"

  mtta:
    name: "Mean Time to Acknowledge"
    definition: "Average time from alert to engineer acknowledgment"
    target: "< 5 minutes"
    trending: "Monitors on-call responsiveness"

  incident_frequency:
    name: "Incidents per week/month by severity"
    target: "Trending downward"
    trending: "Monitors system reliability improvement"

  action_item_completion:
    name: "Post-mortem action item completion rate"
    target: "> 90% completed on time"
    trending: "Monitors follow-through on improvements"

  recurring_incidents:
    name: "Percentage of incidents with same root cause as previous incident"
    target: 
     

Incident Management Checklist

incident_management_checklist:
  process_setup:
    - [ ] Severity levels defined with clear criteria
    - [ ] Escalation matrix documented
    - [ ] On-call schedule established and staffed
    - [ ] War room procedures documented
    - [ ] Post-mortem template created
    - [ ] Communication templates prepared (status page, email)
    - [ ] Incident management tool configured (PagerDuty, OpsGenie)

  per_incident:
    - [ ] Incident declared with severity level
    - [ ] War room established (SEV1/2)
    - [ ] Roles      
     [ ]    
     [ ]     
     [ ]     
     [ ]    
     [ ]  
     [ ]    
     [ ]     

  
     [ ]     
     [ ]      
     [ ]   
     [ ]      
     [ ]     
     [ ]      

Best Practices

• Define severity levels with concrete examples so there is no ambiguity during an active incident
• Implement time-based escalation: if the on-call does not acknowledge, automatically escalate
• Focus on mitigation first, root cause second: restore service before investigating why it failed
• Run blameless post-mortems: the goal is to improve systems, not to assign fault to individuals
• Track post-mortem action items to completion: an unfinished action item means the same incident can recur
• Monitor incident metrics (MTTR, MTTD, frequency) as leading indicators of system reliability
• Protect on-call health: track page volume per person and redistribute if someone is overburdened
• Separate the incident commander role from the technical lead role in SEV1/2 incidents
• Practice incident response regularly with game days or chaos engineering exercises
• Archive incident records and post-mortems for compliance evidence and organizational learning