databricks-incident-runbook

name	databricks-incident-runbook
description	Execute Databricks incident response procedures with triage, mitigation, and postmortem. Use when responding to Databricks-related outages, investigating job failures, or running post-incident reviews for pipeline failures. Trigger with phrases like "databricks incident", "databricks outage", "databricks down", "databricks on-call", "databricks emergency", "job failed".
allowed-tools	Read, Grep, Bash(databricks:*), Bash(curl:*)
version	1.0.0
license	MIT
author	Jeremy Longshore <[e-mail]>
compatible-with	claude-code, codex, openclaw
tags	["saas","databricks","incident-response"]

Databricks Incident Runbook

Overview

Rapid incident response for Databricks: triage script, decision tree, immediate actions by error type, communication templates, evidence collection, and postmortem template. Designed for on-call engineers to follow during live incidents.

Severity Levels

Level	Definition	Response Time	Examples
P1	Production pipeline down	< 15 min	Critical ETL failed, data not updating
P2	Degraded performance	< 1 hour	Slow queries, partial failures, stale data
P3	Non-critical issues	< 4 hours	Dev cluster issues, non-critical job delays
P4	No user impact	Next business day	Monitoring gaps, cleanup needed

Instructions

Step 1: Quick Triage (Run First)

#!/bin/bash
set -euo pipefail
echo "=== DATABRICKS TRIAGE $(date -u +%H:%M:%S\ UTC) ==="

# 1. Is Databricks itself down?
echo "--- Platform Status ---"
curl -s https://status.databricks.com/api/v2/status.json | \
  jq -r '.status.description // "UNKNOWN"'

# 2. Can we reach the workspace?
echo "--- Workspace ---"
if databricks current-user me --output json 2>/dev/null | jq -r .userName; then
    echo "API: CONNECTED"
else
    echo "API: UNREACHABLE — check VPN/firewall/token"
fi

# 3. Recent failures
echo "--- Failed Runs (last 1h) ---"
databricks runs list --limit 20 --output json 2>/dev/null | \
  jq -r '.runs[]? | select(.state.result_state == "FAILED") |
    "\(.run_id): \(.run_name // "unnamed") — \(.state.state_message // "no message")"' || \
   


 
databricks clusters list --output json 2>/dev/null | \
  jq -r  || \