| name | databricks-common-errors |
| description | Diagnose and fix Databricks common errors and exceptions.
Use when encountering Databricks errors, debugging failed jobs,
or troubleshooting cluster and notebook issues.
Trigger with phrases like "databricks error", "fix databricks",
"databricks not working", "debug databricks", "spark error".
|
| allowed-tools | Read, Grep, Bash(databricks:*) |
| version | 1.0.0 |
| license | MIT |
| author | Jeremy Longshore <jeremy@intentsolutions.io> |
| compatible-with | claude-code, codex, openclaw |
| tags | ["saas","databricks","debugging"] |
Databricks Common Errors
Overview
Quick-reference diagnostic guide for the most frequent Databricks errors. Covers cluster failures, Spark OOM, Delta Lake conflicts, permissions, schema mismatches, rate limits, and job run failures with real SDK/SQL solutions.
Prerequisites
- Databricks CLI configured
- Access to cluster/job logs
databricks-sdk installed for programmatic debugging
Instructions
Step 1: Identify the Error Source
databricks runs get --run-id $RUN_ID --output json | jq '{
state: .state.result_state,
message: .state.state_message,
tasks: [.tasks[] | {key: .task_key, state: .state.result_state, error: .state.state_message}]
}'
Step 2: Match and Fix
CLUSTER_NOT_READY / INVALID_STATE
ClusterNotReadyException: Cluster 0123-456789-abcde is not in a RUNNING state
Cause: Cluster is starting, terminating, or in error state.
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.compute import State
w = WorkspaceClient()
cluster = w.clusters.get(cluster_id="0123-456789-abcde")
if cluster.state in (State.PENDING, State.RESTARTING):
w.clusters.ensure_cluster_is_running("0123-456789-abcde")
elif cluster.state == State.TERMINATED:
w.clusters.start_and_wait(cluster_id="0123-456789-abcde")
elif cluster.state == State.ERROR:
reason = cluster.termination_reason
print(f"Cluster error: {reason.code} — {reason.parameters}")
SPARK_DRIVER_OOM
java.lang.OutOfMemoryError: Java heap space
SparkException: Job aborted due to stage failure
Cause: Driver or executor running out of memory.
