Use during incidents to coordinate triage, assign roles, manage communications, track mitigations, run post-incident review, and keep responders aligned under pressure.
Use during incidents to coordinate triage, assign roles, manage communications, track mitigations, run post-incident review, and keep responders aligned under pressure.
zh_description
用于事故响应中的分诊、角色分配、沟通协调、缓解跟踪和复盘。
version
1.0.0
author
seaworld008
source
in-house
source_url
tags
["commander", "devops", "incident", "sre"]
created_at
2026-03-04
updated_at
2026-06-29
quality
5
complexity
intermediate
Incident Commander Skill
Category: Engineering Team Tier: POWERFUL Author: Claude Skills Team Version: 1.0.0 Last Updated: February 2026
Overview
The Incident Commander skill provides a comprehensive incident response framework for managing technology incidents from detection through resolution and post-incident review. This skill implements battle-tested practices from SRE and DevOps teams at scale, providing structured tools for severity classification, timeline reconstruction, and thorough post-incident analysis.
Key Features
Automated Severity Classification - Intelligent incident triage based on impact and urgency metrics
Timeline Reconstruction - Transform scattered logs and events into coherent incident narratives
Post-Incident Review Generation - Structured PIRs with multiple RCA frameworks
Communication Templates - Pre-built templates for stakeholder updates and escalations
Runbook Integration - Generate actionable runbooks from incident patterns
Skills Included
Core Tools
Incident Classifier (incident_classifier.py)
Analyzes incident descriptions and outputs severity levels
Recommends response teams and initial actions
Generates communication templates based on severity
Subject: URGENT - Customer-Impacting Outage - {Service Name}
Executive Summary:
{2-3 sentence description of customer impact and business implications}
Key Metrics:
- Time to Detection: {X minutes}
- Time to Engagement: {X minutes}
- Estimated Customer Impact: {number/percentage}
- Current Status: {status}
- ETA to Resolution: {time or "investigating"}
Leadership Actions Required:
- [ ] Customer communication approval
- [ ] PR/Communications coordination
- [ ] Resource allocation decisions
- [ ] External vendor engagement
Incident Commander: {name} ({contact})
Next Update: {time}
---
This is an automated alert from our incident response system.
Customer Communication Template
We are currently experiencing {brief description of issue} affecting {scope of impact}.
Our engineering team was alerted at {time} and is actively working to resolve the issue. We will provide updates every {frequency} until resolved.
What we know:
- {factual statement of impact}
- {factual statement of scope}
- {brief status of response}
What we're doing:
- {primary response action}
- {secondary response action}
Workaround (if available):
{workaround steps or "No workaround currently available"}
We apologize for the inconvenience and will share more information as it becomes available.
Next update: {time}
Status page: {link}
Stakeholder Management
Stakeholder Classification
Internal Stakeholders:
Engineering Leadership - Technical decisions and resource allocation
Product Management - Customer impact assessment and feature implications
Customer Support - User communication and support ticket management
Sales/Account Management - Customer relationship management for enterprise clients
Executive Team - Business impact decisions and external communication approval
Legal/Compliance - Regulatory reporting and liability assessment
External Stakeholders:
Customers - Service availability and impact communication
Partners - API availability and integration impacts
Vendors - Third-party service dependencies and support escalation
Regulators - Compliance reporting for regulated industries
Public/Media - Transparency for public-facing outages
Communication Cadence by Stakeholder
Stakeholder
SEV1
SEV2
SEV3
SEV4
Engineering Leadership
Real-time
30min
4hrs
Daily
Executive Team
15min
1hr
EOD
Weekly
Customer Support
Real-time
30min
2hrs
As needed
Customers
15min
1hr
Optional
None
Partners
30min
2hrs
Optional
None
Runbook Generation Framework
Dynamic Runbook Components
Detection Playbooks
Monitoring alert definitions
Triage decision trees
Escalation trigger points
Initial response actions
Response Playbooks
Step-by-step mitigation procedures
Rollback instructions
Validation checkpoints
Communication checkpoints
Recovery Playbooks
Service restoration procedures
Data consistency checks
Performance validation
User notification processes
Runbook Template Structure
# {Service/Component} Incident Response Runbook## Quick Reference-**Severity Indicators:** {list of conditions for each severity level}
-**Key Contacts:** {on-call rotations and escalation paths}
-**Critical Commands:** {list of emergency commands with descriptions}
## Detection### Monitoring Alerts- {Alert name}: {description and thresholds}
- {Alert name}: {description and thresholds}
### Manual Detection Signs- {Symptom}: {what to look for and where}
- {Symptom}: {what to look for and where}
## Initial Response (0-15 minutes)1.**Assess Severity** - [ ] Check {primary metric}
- [ ] Verify {secondary indicator}
- [ ] Classify as SEV{level} based on {criteria}
2.**Establish Command** - [ ] Page Incident Commander if SEV1/2
- [ ] Create incident tracking ticket
- [ ] Join war room: {link/bridge info}
3.**Initial Investigation** - [ ] Check recent deployments: {deployment log location}
- [ ] Review error logs: {log location and queries}
- [ ] Verify dependencies: {dependency check commands}
## Mitigation Strategies### Strategy 1: {Name}**Use when:** {conditions}
**Steps:**1. {detailed step with commands}
2. {detailed step with expected outcomes}
3. {validation step}
**Rollback Plan:**1. {rollback step}
2. {verification step}
### Strategy 2: {Name}
{similar structure}
## Recovery and Validation1.**Service Restoration** - [ ] {restoration step}
- [ ] Wait for {metric} to return to normal
- [ ] Validate end-to-end functionality
2.**Communication** - [ ] Update status page
- [ ] Notify stakeholders
- [ ] Schedule PIR
## Common Pitfalls-**{Pitfall}:** {description and how to avoid}
-**{Pitfall}:** {description and how to avoid}
## Reference Information-**Architecture Diagram:** {link}
-**Monitoring Dashboard:** {link}
-**Related Runbooks:** {links to dependent service runbooks}
Post-Incident Review (PIR) Framework
PIR Timeline and Ownership
Timeline:
24 hours: Initial PIR draft completed by Incident Commander
3 business days: Final PIR published with all stakeholder input
1 week: Action items assigned with owners and due dates
4 weeks: Follow-up review on action item progress
Roles:
PIR Owner: Incident Commander (can delegate writing but owns completion)
Technical Contributors: All engineers involved in response
Review Committee: Engineering leadership, affected product teams
Action Item Owners: Assigned based on expertise and capacity
Root Cause Analysis Frameworks
1. Five Whys Method
The Five Whys technique involves asking "why" repeatedly to drill down to root causes:
Example Application:
Problem: Database became unresponsive during peak traffic
Why 1: Why did the database become unresponsive? → Connection pool was exhausted
Why 2: Why was the connection pool exhausted? → Application was creating more connections than usual
Why 3: Why was the application creating more connections? → New feature wasn't properly connection pooling
Why 4: Why wasn't the feature properly connection pooling? → Code review missed this pattern
Why 5: Why did code review miss this? → No automated checks for connection pooling patterns
Best Practices:
Ask "why" at least 3 times, often need 5+ iterations
Focus on process failures, not individual blame
Each "why" should point to a actionable system improvement
Consider multiple root cause paths, not just one linear chain
2. Fishbone (Ishikawa) Diagram
Systematic analysis across multiple categories of potential causes:
Medium: User experience degradation, support ticket volume
Low: Internal tools, development impact only
Escalation Matrix:
Severity
Duration
Business Escalation
SEV1
Immediate
VP Engineering
SEV1
30 minutes
CTO + Customer Success VP
SEV1
1 hour
CEO + Full Executive Team
SEV2
2 hours
VP Engineering
SEV2
4 hours
CTO
SEV3
1 business day
Engineering Manager
Status Page Management
Update Principles
Transparency: Provide factual information without speculation
Timeliness: Update within committed timeframes
Clarity: Use customer-friendly language, avoid technical jargon
Completeness: Include impact scope, status, and next update time
Status Categories
Operational: All systems functioning normally
Degraded Performance: Some users may experience slowness
Partial Outage: Subset of features unavailable
Major Outage: Service unavailable for most/all users
Under Maintenance: Planned maintenance window
Update Template
{Timestamp} - {Status Category}
{Brief description of current state}
Impact: {who is affected and how}
Cause: {root cause if known, "under investigation" if not}
Resolution: {what's being done to fix it}
Next update: {specific time}
We apologize for any inconvenience this may cause.
Action Item Framework
Action Item Categories
Immediate Fixes
Critical bugs discovered during incident
Security vulnerabilities exposed
Data integrity issues
Process Improvements
Communication gaps
Escalation procedure updates
Runbook additions/updates
Technical Debt
Architecture improvements
Monitoring enhancements
Automation opportunities
Organizational Changes
Team structure adjustments
Training requirements
Tool/platform investments
Action Item Template
**Title:** {Concise description of the action}
**Priority:** {Critical/High/Medium/Low}
**Category:** {Fix/Process/Technical/Organizational}
**Owner:** {Assigned person}
**Due Date:** {Specific date}
**Success Criteria:** {How will we know this is complete}
**Dependencies:** {What needs to happen first}
**Related PIRs:** {Links to other incidents this addresses}
**Description:**
{Detailed description of what needs to be done and why}
**Implementation Plan:**
1. {Step 1}
2. {Step 2}
3. {Validation step}
**Progress Updates:**
- {Date}: {Progress update}
- {Date}: {Progress update}
# Quick classification from stdinecho"API rate limits causing customer API calls to fail" | python scripts/incident_classifier.py --format text
# Build timeline from multiple sources
python scripts/timeline_reconstructor.py --input assets/api_incident_logs.json --detect-phases --gap-analysis
# Generate comprehensive PIR
python scripts/pir_generator.py --incident assets/api_incident_summary.json --rca-method fishbone --action-items
Best Practices
During Incident Response
Maintain Calm Leadership
Stay composed under pressure
Make decisive calls with incomplete information
Communicate confidence while acknowledging uncertainty
Document Everything
All actions taken and their outcomes
Decision rationale, especially for controversial calls
Timeline of events as they happen
Effective Communication
Use clear, jargon-free language
Provide regular updates even when there's no new information
Manage stakeholder expectations proactively
Technical Excellence
Prefer rollbacks to risky fixes under pressure
Validate fixes before declaring resolution
Plan for secondary failures and cascading effects
Post-Incident
Blameless Culture
Focus on system failures, not individual mistakes
Encourage honest reporting of what went wrong
Celebrate learning and improvement opportunities
Action Item Discipline
Assign specific owners and due dates
Track progress publicly
Prioritize based on risk and effort
Knowledge Sharing
Share PIRs broadly within the organization
Update runbooks based on lessons learned
Conduct training sessions for common failure modes
Continuous Improvement
Look for patterns across multiple incidents
Invest in tooling and automation
Regularly review and update processes
Integration with Existing Tools
Monitoring and Alerting
PagerDuty/Opsgenie integration for escalation
Datadog/Grafana for metrics and dashboards
ELK/Splunk for log analysis and correlation
Communication Platforms
Slack/Teams for war room coordination
Zoom/Meet for video bridges
Status page providers (Statuspage.io, etc.)
Documentation Systems
Confluence/Notion for PIR storage
GitHub/GitLab for runbook version control
JIRA/Linear for action item tracking
Change Management
CI/CD pipeline integration
Deployment tracking systems
Feature flag platforms for quick rollbacks
Conclusion
The Incident Commander skill provides a comprehensive framework for managing incidents from detection through post-incident review. By implementing structured processes, clear communication templates, and thorough analysis tools, teams can improve their incident response capabilities and build more resilient systems.
The key to successful incident management is preparation, practice, and continuous learning. Use this framework as a starting point, but adapt it to your organization's specific needs, culture, and technical environment.
Remember: The goal isn't to prevent all incidents (which is impossible), but to detect them quickly, respond effectively, communicate clearly, and learn continuously.
