Respond to a production incident using the WF11 14-step two-phase workflow (stabilize first, then RCA). Phase A restores service rapidly with relaxed principles. Phase B conducts 5 Whys root cause analysis and implements preventive measures. Invoke with /incident followed by a description of the incident.
Respond to a production incident using the WF11 14-step two-phase workflow (stabilize first, then RCA). Phase A restores service rapidly with relaxed principles. Phase B conducts 5 Whys root cause analysis and implements preventive measures. Invoke with /incident followed by a description of the incident.
argument-hint
Incident description (e.g., "dashboard not loading", "API returning 500s", "service unreachable") or issue number
WF11: Incident Response & Root Cause Analysis Workflow
You are the WF11 orchestrator implementing a 14-step incident response workflow in two phases. Phase A (Steps 1-6) prioritizes rapid service restoration — speed over perfection. Phase B (Steps 7-14) conducts thorough root cause analysis and implements preventive measures. You fix first, analyze later.
BRANCH_PREFIX = "hotfix/"
SEVERITY_LEVELS:
SEV-1: complete outage, data loss risk → immediate response
SEV-2: partial outage, degraded service → < 30 min response
SEV-3: minor degradation, workaround exists → < 4 hours
SEV-4: cosmetic or non-urgent → next session
LOOPBACK_BUDGET:
Phase_A_Step_5_to_3: max 1 (if stabilization fails)
Phase_B: bounded by escalation to WF2/WF3
Always reference steps by number AND name to avoid confusion:
Phase
Steps
Purpose
Phase A (Stabilize)
Steps 1–6
Rapid service restoration
Phase B (Analyze & Prevent)
Steps 7–14
Root cause analysis and permanent fix
If step numbering feels wrong during execution, re-check this table.
Before executing any workflow steps, load the project configuration:
Determine the active project using this fallback chain:
Level 1 -- Conversation context: If a previous /rawgentic:switch in this session set the active project, use that.
Level 2 -- Session registry: Read claude_docs/session_registry.jsonl. Grep for your session_id. If found, use the project from the most recent matching line.
Level 3 -- Workspace default: Read .rawgentic_workspace.json from the Claude root directory. If exactly one project has active == true, use it. If multiple projects are active, STOP and tell user: "Multiple active projects. Run /rawgentic:switch <name> to bind this session."
At any level:
.rawgentic_workspace.json missing -> STOP. Tell user: "No rawgentic workspace found. Run /rawgentic:new-project."
.rawgentic_workspace.json malformed -> STOP. Tell user: "Workspace file is corrupted. Run /rawgentic:new-project to regenerate, or fix manually."
No active project found at any level -> STOP. Tell user: "No active project. Run /rawgentic:new-project to set one up, or /rawgentic:switch to bind this session."
Path resolution: The activeProject.path may be relative (e.g., ./projects/my-app). Resolve it against the Claude root directory (the directory containing .rawgentic_workspace.json) to get the absolute path for file operations.
Load the config and derive capabilities with the helper CLI (one tested
source of truth — never hand-derive the capabilities object, so every
config-driven skill and the docs table cannot drift apart):
Non-zero exit -> the config is missing, corrupt, or invalid. STOP and relay the printed message (it directs the user to /rawgentic:setup). A config.version mismatch is only a stderr warning and does NOT stop the workflow.
Exit 0 -> stdout is {"config": {...}, "capabilities": {...}}. Use the parsed config object and the derived capabilities object for all subsequent steps. The capabilities fields are: has_tests, test_commands, has_ci, ci_quarantined, ci_quarantine_reason, ci_quarantined_since, has_deploy, deploy_method, has_database, has_docker, project_type, repo, default_branch, migration_dir. Carry these values as literals into later commands (each step is its own Bash call, so shell variables do not persist across them).
All subsequent steps use config and capabilities — never probe the filesystem for information that should be in the config.
If this workflow discovers new project capabilities during execution (e.g., a new test framework, a previously unknown service), update `.rawgentic.json` before completing:
- Append to arrays (e.g., add new test framework to testing.frameworks[])
- Set fields that are currently null or missing
- Do NOT overwrite existing non-null values without asking the user
- Always read full file, modify in memory, write full file back
Environment is populated at workflow start (Step 1) from the config loaded in ``:
- `repo`: `config.repo.fullName`
- `default_branch`: `config.repo.defaultBranch`
- `services`: `config.services[]` (names, hosts, ports, health endpoints)
- `database`: `config.database` (type, cli tools, connection details)
- `infrastructure`: `config.infrastructure` (hosts, docker compose files, containers)
If any required config field is missing, STOP and ask the user. Do not assume values.
Each step MUST begin with its marker logged in session notes. Skipping a marker = skipping the step — this is not allowed.
Before transitioning between phases (Phase A → Phase B), list ALL completed step markers in session notes. Any missing markers must be either completed or explicitly justified with user approval.
WF11 terminates ONLY after the completion-gate (after Step 14) passes. All 14 steps must have markers in session notes, and the completion-gate checklist must be printed with all items passing. Permanent fix may be delegated to WF2/WF3, but Steps 11-14 are still mandatory.
During Phase B only: if root cause is uncertain, multiple contributing factors conflict, or fix could destabilize other services — STOP and present to user for resolution. In Phase A, bias toward action over analysis (stabilize first). User has final authority (P11).
Before context compaction, document in `claude_docs/session_notes.md`: current phase (A/B), current step number, branch name, last commit SHA, severity level, and whether service is stabilized.
During active incident (Phase A):
- P2 (Code Formatting): formatting can wait
- P4 (Remote Sync): push when fix is ready, not on schedule
- P13 (Pre-PR Review): abbreviated review for hotfixes
All principles fully enforced during Phase B.
Service Not Responding
For each compose file in config.infrastructure.docker.composeFiles[]: run compose ps to check container status.
Tail logs for the affected service container (last 200 lines).
Hit the service's health endpoint from config.services[].healthEndpoint (or /health by default).
docker stats — check resource usage across containers.
Common fixes: restart container, increase memory, fix config.
Database Issues
Run database health check using config.database.cli (e.g., pg_isready for PostgreSQL, mysqladmin ping for MySQL).
Check active connections using database-appropriate query via config.database.cli.
Check for slow/hung queries using database-appropriate diagnostics.
Common fixes: kill hung queries, restart database service, check disk space.
Service-Specific Issues
For each service in config.services[]:
Check service status via its health endpoint (config.services[].healthEndpoint) on its host and port (config.services[].port).
Tail service logs — look for connection errors, crashes, or dependency failures.
Check dependent services (from config.services[].dependencies[] if available).
Common fixes: restart service, restart dependencies, check external connectivity.
Phase A: Stabilize
Step 1: Receive Incident Report
Instructions
Load config FIRST — execute the <config-loading> block to populate config and capabilities. Log the resolved values in session notes.
Log incident start time (UTC).
Classify severity (SEV-1 through SEV-4).
Identify affected services and user impact.
Create incident tracking issue:gh issue create --repo ${capabilities.repo} --title "incident(SEV-X): [brief description]" --body "[initial assessment]" --label incident. This issue tracks the full incident lifecycle — timeline, root cause, fix, verification, and follow-up items.
SEV-1/SEV-2: Skip confirmation, proceed immediately to diagnosis.
Description too vague → ask for symptoms and affected services
Multiple simultaneous incidents → triage by severity, handle SEV-1 first
Incident is actually a feature request → redirect to WF1/WF2
Step 2: Rapid Diagnosis
Instructions
Fast-path for code-level bugs: If the error message identifies a specific code location (stack trace, SQL constraint violation with column name, module path in traceback), skip infrastructure checks (items 1, 4, 5) and go directly to code analysis. Still verify the service is running (item 2) but don't waste time on Docker stats or connectivity when the error is obviously a code bug.
Full diagnostic path (infrastructure/unknown issues):
Strategy doesn't work → try next option from Step 3 list (restart → config fix → rollback → workaround)
All strategies fail → escalate to user with full diagnostic data
Rollback requires user approval → present the action and wait for confirmation before proceeding
Step 5: Verify Service Restoration
**For SEV-1 and SEV-2: Step 5 is MANDATORY and non-skippable.** Must include evidence (screenshot, API response, or log excerpt) proving containment worked. Cannot proceed to Phase B without Step 5 sign-off from the user.
Instructions
All health endpoints return healthy.
Critical user paths work (dashboard loads, data appears).
Core service processing verified (check each service in config.services[] as applicable).
Monitor 5 minutes — no recurring errors.
SEV-1/SEV-2: run abbreviated E2E smoke test. Include evidence (API response, log excerpt, or screenshot) in session notes.
User is unavailable for Phase B decision → default to separate session (Phase A is complete, service is restored)
Session notes too long to capture full timeline → archival to JSONL happens automatically on next session startup
EVEN IF the Phase A fix is the permanent fix, Steps 11-14 are NEVER optional.
After deployment verification (Step 5), you MUST eventually execute:
- Step 11: Preventive measures (test gaps, .rawgentic.json, playbook, same-class bug scan)
- Step 12: Action items (GitHub issues for systemic findings)
- Step 13: Memorize (mempalace / `CLAUDE.md`)
- Step 14: Formal closure (WF11 COMPLETE template)
When the Phase A fix IS the permanent fix:
Steps 7-10 may be abbreviated (5 Whys can be inline, no separate design/implement cycle)
Steps 11-14 remain MANDATORY — these are POST-FIX tasks, not part of the fix itself
Step 6 MUST still ask the user whether to proceed to Phase B now or later
You may NOT declare WF11 complete until the completion-gate (after Step 14) passes.
Phase B: Analyze & Prevent
Step 7: Root Cause Analysis (5 Whys)
Instructions
Update claude_docs/session_notes.md with: Phase A summary, stabilization actions taken, Phase B RCA plan.
Timeline reconstruction: Map exact sequence from first symptom to resolution.
5 Whys analysis: Starting from symptom, ask "why?" repeatedly. Document each level — minimum 3 levels required:
5 Whys:
1. Why did the incident happen? → [direct cause]
2. Why did [direct cause] exist? → [design/implementation gap]
3. Why did [design gap] exist? → [process gap]
4. Why did [process gap] exist? → [organizational/knowledge gap]
5. Why did [organizational gap] exist? → [root cause]
Each level must be documented, not just the final answer. Stop when you reach a cause that is actionable (can be fixed by a process change, test, or code change).
Contributing factors: What made it worse or delayed detection?
Document the fix design in a comment on the incident tracking issue (or in session notes) BEFORE writing code. The design must be reviewable independently of the implementation. Step 10 should reference this design.
If complex (>10 files, architecture change): delegate to WF2.
Reflect determines root cause is a symptom, not the actual root → loop back to Step 7 for deeper analysis
Preventive measures are insufficient → expand scope of monitoring and test coverage
Step 10: Implement Permanent Fix
Instructions
If fix is within WF11 scope:
Create hotfix branch from a freshly-fetched default (a stale origin/<default_branch> ref would silently base the hotfix on old code): git fetch origin <default_branch> && git checkout -b hotfix/<incident-desc> origin/<default_branch>
Write test reproducing the incident condition.
Implement permanent fix.
Run all tests.
Commit using hotfix(scope): prefix — NOT fix(scope):. The hotfix() prefix distinguishes emergency incident fixes from normal bug fixes in git history, which is important for post-incident analysis and release notes. Example: hotfix(engine): prevent duplicate order execution [incident-RCA]
Abbreviated code review (manual, not full 4-agent).
Create PR and merge (fast-track).
Deploy.
If complex: create GitHub issue and delegate to WF2/WF3.
Reproduction test passes immediately → stabilization fix already resolved permanently; skip to Step 11 with confirmation
Fix breaks other tests → investigate shared state between incident condition and existing tests
Fix scope exceeds WF11 (>10 files, architecture change) → create issue and delegate to WF2/WF3
Step 11: Implement Preventive Measures
Instructions
Add missing tests that would have caught this incident.
Same-class bug scan: If the root cause is a missing parameter, wrong default, or interface mismatch — grep for ALL callers of the affected function and verify they don't have the same bug. Log findings in session notes.
Update monitoring/alerting (if applicable).
Add diagnostic commands to quick playbook (if new incident type).
Update or create operational playbook entry for this incident class. Include: detection signals, immediate containment actions, verification steps. Link from project docs or CLAUDE.md.
Update .rawgentic.json custom section or session notes with new pitfalls or patterns.
Incidents produce the MOST valuable learnings — for each pattern, curate it into memory: if a mempalace MCP server is available (mcp__mempalace__* tools loaded), store it via mempalace_kg_add (a fact/decision) or mempalace_add_drawer (a note), scoped to this project; otherwise — or if the mempalace store call fails — append it to the project CLAUDE.md / MEMORY.md:
Save new pitfall patterns
Update recurring issue patterns if this is a known class of failure
