Databricks Cluster Forensics
The operational SRE spine of the pack — what a Databricks engineer reaches for at
2 AM when the compute layer is broken or unexplained. It correlates a cluster's
live event stream across API surfaces to name the failure with its actual error
code and its version-specific mitigation, not "network problem, try again".
Overview
Six real compute-layer failures live in this skill; each has a deterministic
detector and an on-demand reference:
- Cold-start long-tail (D01) — on VNet/VPC-injected workspaces a 5-minute
start randomly takes 20-35, and Databricks reports only the aggregate.
scripts/cluster-coldstart-forensics.py splits the PENDING window into stages
(provisioning / init-scripts / spark-startup) so you see which stage spiked.
- Photon premium without the speedup (D02) — Photon silently falls back to
Spark on UDFs while the cluster still bills the ~2× Photon DBU premium for its
whole uptime. See
references/photon-eligibility-and-fallback.md.
- DBR upgrade landmines (D03/D04/D05) — 14.x moved the working dir to the
workspace filesystem (a ~500 MB cap that silently breaks large intermediate
writes); 15.1 removed DBFS-root library storage and JDK 11; 15.4 flipped a JDBC
calendar default.
scripts/find-cwd-writes.py (AST) and scripts/scan-jar-jdk.sh
(bytecode target) are the pre-upgrade detectors; references/dbr-upgrade-paths.md
is the per-hop encyclopedia.
- The launch-failure umbrella (D06) —
CLOUD_PROVIDER_LAUNCH_FAILURE /
NPIP_TUNNEL_SETUP_FAILURE each hide five distinct causes (subnet IP
exhaustion, DNS, NSG/security-group block, deleted VNet, cloud throttling).
references/termination-codes.md disambiguates them.
- Spot shuffle aborts (D10) — a reclaimed spot node forces a shuffle
recompute; lose another mid-recompute and the stage exceeds
spark.stage.maxConsecutiveAttempts and the job aborts.
references/spot-vs-ondemand-decision.md is the config decision tree.
It is architecturally distinct from the v1 databricks-common-errors and
databricks-incident-runbook skills: those narrate. This one reads live cluster
events, buckets them deterministically (the arithmetic is in scripts/, never
eyeballed), fans out parallel root-cause threads via the cluster-event-investigator
subagent, and loads deep knowledge from references/ only when a symptom needs it.
Two data planes. Cluster control-plane evidence (spec, state, event stream)
comes from the custom databricks-workspace-mcp (clusters_get /
clusters_events / clusters_list). The Photon audit's system.query.history
read runs through the CLI Statement Execution API (databricks api post /api/2.0/sql/statements) — the same path databricks-cost-leak-hunter uses.
Either surface absent, the skill degrades to advisory mode and accepts pasted
event JSON / query plans so it still produces value.
Prerequisites
databricks-workspace-mcp registered — the source of clusters_get,
clusters_events, clusters_list. Absent, the skill accepts a pasted
clusters.events response and says so (advisory mode).- Databricks CLI authenticated (
databricks auth login, or the
DATABRICKS_HOST + DATABRICKS_TOKEN env pair) and jq — for the Photon
system.query.history read.
DATABRICKS_WAREHOUSE_ID set to a running SQL warehouse — required only for
the Photon audit (Step 2); the cold-start / launch-failure flows need only the
workspace MCP.unzip (and ideally a JDK's javap) on PATH for the DBR-15.1 JAR scan
(scan-jar-jdk.sh falls back to reading class-file bytes if javap is absent).
The skill checks which surfaces are present in Step 0 and reports what is missing
before starting a flow it cannot finish.
Instructions
Pick the flow by symptom. Each is independent; run only what the question needs.
Step 0: Detect Available Surfaces
Confirm the workspace MCP answers (clusters_list returns) and, for a Photon
audit, that the CLI is authenticated and DATABRICKS_WAREHOUSE_ID is set. Name
any missing surface and switch that flow to advisory mode (pasted input) rather
than failing mid-diagnosis.
Step 1: Cold-Start / Launch-Failure Forensics (D01, D06)
Pull the cluster's event stream and bucket its PENDING time:
databricks clusters events --cluster-id "$CLUSTER_ID" --output json > "$OUT/events.json"
python3 "${CLAUDE_SKILL_DIR}/scripts/cluster-coldstart-forensics.py" \
--input "$OUT/events.json"
- If the start succeeded but was slow, the dominant stage names the layer:
provisioning → cloud VM allocation or network/DNS/NPIP; init-scripts → a slow
init script or library install; spark-startup → driver spin-up.
- If the start failed, read the terminal
termination_reason.code and
disambiguate with
${CLAUDE_SKILL_DIR}/references/termination-codes.md
— especially the CLOUD_PROVIDER_LAUNCH_FAILURE / NPIP_TUNNEL_SETUP_FAILURE
umbrella and its five sub-causes.
For a messy failure, hand the cluster_id to the cluster-event-investigator
subagent (/investigate-cluster <id>): it fans out one thread per cause class and
returns the single most-likely cause with its evidence.
Step 2: Photon Fallback Audit (D02)
Check whether Photon is earning its premium. Query recent query history for plans
that fell back to Spark, then corroborate the cluster is Photon (runtime_engine
via clusters_get):
databricks api post /api/2.0/sql/statements --json "$(jq -n --arg wh "$DATABRICKS_WAREHOUSE_ID" \
'{warehouse_id:$wh, wait_timeout:"30s",
statement:"SELECT statement_id, executed_by, total_duration_ms FROM system.query.history WHERE end_time > now() - INTERVAL 1 DAY ORDER BY total_duration_ms DESC LIMIT 50"}')"
Then read the physical plan of the slow statements for the "Photon does not
support" seam and the ColumnarToRow / RowToColumnar boundaries — the detection
recipe and the UDF-rewrite fixes are in
${CLAUDE_SKILL_DIR}/references/photon-eligibility-and-fallback.md.
Step 3: DBR Upgrade Readiness (D03, D04, D05)
Before bumping the runtime, run the two pre-upgrade detectors against the job's
code and libraries:
python3 "${CLAUDE_SKILL_DIR}/scripts/find-cwd-writes.py" --risk-only path/to/job/
bash "${CLAUDE_SKILL_DIR}/scripts/scan-jar-jdk.sh" path/to/libs/
Cross-reference each hop's landmines (the 14.x CWD cap, the 15.1 DBFS-root-library
and JDK-11 removals, the 15.4 JDBC calendar flip) in
${CLAUDE_SKILL_DIR}/references/dbr-upgrade-paths.md.
Step 4: Spot Configuration Review (D10)
If a job keeps aborting after NODES_LOST / SPOT_INSTANCE_TERMINATION around a
shuffle, read the cluster's aws_attributes (clusters_get) and check the
driver-on-demand rule and the spot ratio against
${CLAUDE_SKILL_DIR}/references/spot-vs-ondemand-decision.md.
The #1 fix is pinning the driver (and a floor of workers) to on-demand so a spot
reclaim can never take the driver.
Output
- A cold-start stage breakdown — total PENDING time split into
provisioning / init-scripts / spark-startup, the dominant stage named, and the
layer to investigate (or, for a failed start, the terminal code + its cause).
- A root-cause verdict (from
cluster-event-investigator) — the single
most-likely cause with the specific events/codes that point to it, and the
cause classes ruled out.
- A Photon audit — the queries paying the premium while falling back to Spark,
with the plan seam and the UDF-rewrite fix.
- A DBR-upgrade risk list — the CWD writes at risk under the 500 MB cap and
the JARs built for a pre-17 JDK, each with its line/file, plus the per-hop
breaking-change notes.
- A spot recommendation — the corrected
aws_attributes (driver on-demand,
spot ratio) for the job class.
Error Handling
| Error | Cause | Solution |
|---|
NPIP_TUNNEL_SETUP_FAILURE / CLOUD_PROVIDER_LAUNCH_FAILURE | One of five sub-causes (IP exhaustion, DNS, NSG, deleted VNet, throttling) | Disambiguate via termination-codes.md; the fix differs per sub-cause — do not blanket-retry. |
clusters_events empty or truncated | Databricks prunes old events | Note the truncation; a missing INIT_SCRIPTS_FINISHED may mean "pruned", not "hung" — do not infer an init-script hang from absence alone. |
| Workspace MCP not registered | Connector not set up | Advisory mode: accept a pasted clusters.events JSON and run the forensics script on it. |
| Photon audit returns nothing | No system.query.history grant, or DATABRICKS_WAREHOUSE_ID unset | Confirm the warehouse id and the system.query grant chain; degrade to reading a pasted query plan. |
scan-jar-jdk.sh reports JDK ? | JAR has no class files, or unzip missing | Install unzip; a JDK ? means the JAR is resources-only (no bytecode to check). |
| Cold-start script says "unmeasured" for a stage | The boundary events are absent (no init scripts, or pruned events) | Expected — the script never folds an unmeasured stage into another; investigate the measured stages. |
Examples
Example 1: "My cluster randomly takes 25 minutes to start."
Step 1 buckets the events: provisioning 21m (84%), init-scripts 1m, spark-startup 3m. Dominant is provisioning → the skill points at cloud VM
allocation / subnet-IP / DNS, not init scripts, and loads termination-codes.md
for the provisioning sub-causes to check.
Example 2: "Cluster failed with NPIP_TUNNEL_SETUP_FAILURE."
The investigator subagent runs its threads; the network/NPIP thread owns it and
disambiguates to "custom DNS could not resolve the control-plane hostname" (vs the
other four causes), citing the exact check from termination-codes.md.
Example 3: "We're upgrading DBR 13.3 → 15.4. What breaks?"
Step 3 runs find-cwd-writes.py (flags 3 to_parquet("staging/…") writes at risk
under the 14.x cap) and scan-jar-jdk.sh (flags 2 JARs built for JDK 11), and
dbr-upgrade-paths.md surfaces the 15.4 JDBC calendar flip for the pipeline's
pre-Gregorian date handling.
Example 4: "Job keeps dying after losing spot nodes."
Step 4 reads aws_attributes, finds the driver is on spot, and recommends
first_on_demand covering the driver + a worker floor with SPOT_WITH_FALLBACK,
citing the shuffle-recompute cascade in spot-vs-ondemand-decision.md.
Resources
