Diagnose broken or unexplained Databricks compute — slow cold starts, failed
cluster launches, Photon paying its premium without the speedup, DBR-upgrade
landmines, and spot-interruption shuffle aborts — by correlating a cluster's
live event stream across API surfaces. Use when a Databricks cluster won't
start, died mid-run, is randomly slow to start, when planning a Databricks
Runtime upgrade, or when a job keeps failing on spot loss. Trigger with
"databricks cluster won't start", "cluster failed", "why is my cluster slow",
"NPIP_TUNNEL_SETUP_FAILURE", "databricks runtime upgrade", "photon not helping".
Instrucciones de origen · Vista previa de solo lectura
name
databricks-cluster-forensics
description
Diagnose broken or unexplained Databricks compute — slow cold starts, failed
cluster launches, Photon paying its premium without the speedup, DBR-upgrade
landmines, and spot-interruption shuffle aborts — by correlating a cluster's
live event stream across API surfaces. Use when a Databricks cluster won't
start, died mid-run, is randomly slow to start, when planning a Databricks
Runtime upgrade, or when a job keeps failing on spot loss. Trigger with
"databricks cluster won't start", "cluster failed", "why is my cluster slow",
"NPIP_TUNNEL_SETUP_FAILURE", "databricks runtime upgrade", "photon not helping".
The operational SRE spine of the pack — what a Databricks engineer reaches for at
2 AM when the compute layer is broken or unexplained. It correlates a cluster's
live event stream across API surfaces to name the failure with its actual error
code and its version-specific mitigation, not "network problem, try again".
Overview
Six real compute-layer failures live in this skill; each has a deterministic
detector and an on-demand reference:
Cold-start long-tail (D01) — on VNet/VPC-injected workspaces a 5-minute
start randomly takes 20-35, and Databricks reports only the aggregate.
scripts/cluster-coldstart-forensics.py splits the PENDING window into stages
(provisioning / init-scripts / spark-startup) so you see which stage spiked.
Photon premium without the speedup (D02) — Photon silently falls back to
Spark on UDFs while the cluster still bills the ~2× Photon DBU premium for its
whole uptime. See references/photon-eligibility-and-fallback.md.
DBR upgrade landmines (D03/D04/D05) — 14.x moved the working dir to the
workspace filesystem (a ~500 MB cap that silently breaks large intermediate
writes); 15.1 removed DBFS-root library storage and JDK 11; 15.4 flipped a JDBC
calendar default. scripts/find-cwd-writes.py (AST) and scripts/scan-jar-jdk.sh
(bytecode target) are the pre-upgrade detectors; references/dbr-upgrade-paths.md
is the per-hop encyclopedia.
The launch-failure umbrella (D06) — CLOUD_PROVIDER_LAUNCH_FAILURE /
NPIP_TUNNEL_SETUP_FAILURE each hide five distinct causes (subnet IP
exhaustion, DNS, NSG/security-group block, deleted VNet, cloud throttling).
references/termination-codes.md disambiguates them.
Spot shuffle aborts (D10) — a reclaimed spot node forces a shuffle
recompute; lose another mid-recompute and the stage exceeds
spark.stage.maxConsecutiveAttempts and the job aborts.
references/spot-vs-ondemand-decision.md is the config decision tree.
It is architecturally distinct from the v1 databricks-common-errors and
databricks-incident-runbook skills: those narrate. This one reads live cluster
events, buckets them deterministically (the arithmetic is in scripts/, never
eyeballed), fans out parallel root-cause threads via the cluster-event-investigator
subagent, and loads deep knowledge from references/ only when a symptom needs it.
Two data planes. Cluster control-plane evidence (spec, state, event stream)
comes from the custom databricks-workspace-mcp (clusters_get /
clusters_events / clusters_list). The Photon audit's system.query.history
read runs through the CLI Statement Execution API (databricks api post /api/2.0/sql/statements) — the same path databricks-cost-leak-hunter uses.
Either surface absent, the skill degrades to advisory mode and accepts pasted
event JSON / query plans so it still produces value.
Prerequisites
databricks-workspace-mcp registered — the source of clusters_get,
clusters_events, clusters_list. Absent, the skill accepts a pasted
clusters.events response and says so (advisory mode).
Databricks CLI authenticated (databricks auth login, or the
DATABRICKS_HOST + DATABRICKS_TOKEN env pair) and jq — for the Photon
system.query.history read.
DATABRICKS_WAREHOUSE_ID set to a running SQL warehouse — required only for
the Photon audit (Step 2); the cold-start / launch-failure flows need only the
workspace MCP.
unzip (and ideally a JDK's javap) on PATH for the DBR-15.1 JAR scan
(scan-jar-jdk.sh falls back to reading class-file bytes if javap is absent).
The skill checks which surfaces are present in Step 0 and reports what is missing
before starting a flow it cannot finish.
Instructions
Pick the flow by symptom. Each is independent; run only what the question needs.
Step 0: Detect Available Surfaces
Confirm the workspace MCP answers (clusters_list returns) and, for a Photon
audit, that the CLI is authenticated and DATABRICKS_WAREHOUSE_ID is set. Name
any missing surface and switch that flow to advisory mode (pasted input) rather
than failing mid-diagnosis.
Pull the cluster's event stream and bucket its PENDING time:
# events from the workspace MCP (clusters_events) or the CLI, saved to a file:
databricks clusters events --cluster-id "$CLUSTER_ID" --output json > "$OUT/events.json"
python3 "${CLAUDE_SKILL_DIR}/scripts/cluster-coldstart-forensics.py" \
--input "$OUT/events.json"
If the start succeeded but was slow, the dominant stage names the layer:
provisioning → cloud VM allocation or network/DNS/NPIP; init-scripts → a slow
init script or library install; spark-startup → driver spin-up.
If the start failed, read the terminal termination_reason.code and
disambiguate with
${CLAUDE_SKILL_DIR}/references/termination-codes.md
— especially the CLOUD_PROVIDER_LAUNCH_FAILURE / NPIP_TUNNEL_SETUP_FAILURE
umbrella and its five sub-causes.
For a messy failure, hand the cluster_id to the cluster-event-investigator
subagent (/investigate-cluster <id>): it fans out one thread per cause class and
returns the single most-likely cause with its evidence.
Step 2: Photon Fallback Audit (D02)
Check whether Photon is earning its premium. Query recent query history for plans
that fell back to Spark, then corroborate the cluster is Photon (runtime_engine
via clusters_get):
databricks api post /api/2.0/sql/statements --json "$(jq -n --arg wh "$DATABRICKS_WAREHOUSE_ID" \
'{warehouse_id:$wh, wait_timeout:"30s",
statement:"SELECT statement_id, executed_by, total_duration_ms FROM system.query.history WHERE end_time > now() - INTERVAL 1 DAY ORDER BY total_duration_ms DESC LIMIT 50"}')"
Then read the physical plan of the slow statements for the "Photon does not
support" seam and the ColumnarToRow / RowToColumnar boundaries — the detection
recipe and the UDF-rewrite fixes are in
${CLAUDE_SKILL_DIR}/references/photon-eligibility-and-fallback.md.
Step 3: DBR Upgrade Readiness (D03, D04, D05)
Before bumping the runtime, run the two pre-upgrade detectors against the job's
code and libraries:
# D03 — writes to the CWD that the DBR-14 500 MB workspace-FS cap will break:
python3 "${CLAUDE_SKILL_DIR}/scripts/find-cwd-writes.py" --risk-only path/to/job/
# D04 — JARs built for a pre-17 JDK that DBR 15.1's JDK 17 may reject at runtime:
bash "${CLAUDE_SKILL_DIR}/scripts/scan-jar-jdk.sh" path/to/libs/
If a job keeps aborting after NODES_LOST / SPOT_INSTANCE_TERMINATION around a
shuffle, read the cluster's aws_attributes (clusters_get) and check the
driver-on-demand rule and the spot ratio against
${CLAUDE_SKILL_DIR}/references/spot-vs-ondemand-decision.md.
The #1 fix is pinning the driver (and a floor of workers) to on-demand so a spot
reclaim can never take the driver.
Output
A cold-start stage breakdown — total PENDING time split into
provisioning / init-scripts / spark-startup, the dominant stage named, and the
layer to investigate (or, for a failed start, the terminal code + its cause).
A root-cause verdict (from cluster-event-investigator) — the single
most-likely cause with the specific events/codes that point to it, and the
cause classes ruled out.
A Photon audit — the queries paying the premium while falling back to Spark,
with the plan seam and the UDF-rewrite fix.
A DBR-upgrade risk list — the CWD writes at risk under the 500 MB cap and
the JARs built for a pre-17 JDK, each with its line/file, plus the per-hop
breaking-change notes.
A spot recommendation — the corrected aws_attributes (driver on-demand,
spot ratio) for the job class.
One of five sub-causes (IP exhaustion, DNS, NSG, deleted VNet, throttling)
Disambiguate via termination-codes.md; the fix differs per sub-cause — do not blanket-retry.
clusters_events empty or truncated
Databricks prunes old events
Note the truncation; a missing INIT_SCRIPTS_FINISHED may mean "pruned", not "hung" — do not infer an init-script hang from absence alone.
Workspace MCP not registered
Connector not set up
Advisory mode: accept a pasted clusters.events JSON and run the forensics script on it.
Photon audit returns nothing
No system.query.history grant, or DATABRICKS_WAREHOUSE_ID unset
Confirm the warehouse id and the system.query grant chain; degrade to reading a pasted query plan.
scan-jar-jdk.sh reports JDK ?
JAR has no class files, or unzip missing
Install unzip; a JDK ? means the JAR is resources-only (no bytecode to check).
Cold-start script says "unmeasured" for a stage
The boundary events are absent (no init scripts, or pruned events)
Expected — the script never folds an unmeasured stage into another; investigate the measured stages.
Examples
Example 1: "My cluster randomly takes 25 minutes to start."
Step 1 buckets the events: provisioning 21m (84%), init-scripts 1m, spark-startup 3m. Dominant is provisioning → the skill points at cloud VM
allocation / subnet-IP / DNS, not init scripts, and loads termination-codes.md
for the provisioning sub-causes to check.
Example 2: "Cluster failed with NPIP_TUNNEL_SETUP_FAILURE."
The investigator subagent runs its threads; the network/NPIP thread owns it and
disambiguates to "custom DNS could not resolve the control-plane hostname" (vs the
other four causes), citing the exact check from termination-codes.md.
Example 3: "We're upgrading DBR 13.3 → 15.4. What breaks?"
Step 3 runs find-cwd-writes.py (flags 3 to_parquet("staging/…") writes at risk
under the 14.x cap) and scan-jar-jdk.sh (flags 2 JARs built for JDK 11), and
dbr-upgrade-paths.md surfaces the 15.4 JDBC calendar flip for the pipeline's
pre-Gregorian date handling.
Example 4: "Job keeps dying after losing spot nodes."
Step 4 reads aws_attributes, finds the driver is on spot, and recommends
first_on_demand covering the driver + a worker floor with SPOT_WITH_FALLBACK,
citing the shuffle-recompute cascade in spot-vs-ondemand-decision.md.
