databricks-common-errors

name	databricks-common-errors
description	Diagnose and fix Databricks common errors and exceptions. Use when encountering Databricks errors, debugging failed jobs, or troubleshooting cluster and notebook issues. Trigger with phrases like "databricks error", "fix databricks", "databricks not working", "debug databricks", "spark error".
allowed-tools	Read, Grep, Bash(databricks:*)
version	1.0.0
license	MIT
author	Jeremy Longshore <[e-mail]>
compatible-with	claude-code, codex, openclaw
tags	["saas","databricks","debugging"]

Databricks Common Errors

Overview

Quick-reference diagnostic guide for the most frequent Databricks errors. Covers cluster failures, Spark OOM, Delta Lake conflicts, permissions, schema mismatches, rate limits, and job run failures with real SDK/SQL solutions.

Prerequisites

• Databricks CLI configured
• Access to cluster/job logs
• databricks-sdk installed for programmatic debugging

Instructions

Step 1: Identify the Error Source

# Get failed run details
databricks runs get --run-id $RUN_ID --output json | jq '{
  state: .state.result_state,
  message: .state.state_message,
  tasks: [.tasks[] | {key: .task_key, state: .state.result_state, error: .state.state_message}]
}'

Step 2: Match and Fix

---

CLUSTER_NOT_READY / INVALID_STATE

ClusterNotReadyException: Cluster 0123-456789-abcde is not in a RUNNING state

Cause: Cluster is starting, terminating, or in error state.

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.compute import State

w = WorkspaceClient()
cluster = w.clusters.get(cluster_id="0123-456789-abcde")

if cluster.state in (State.PENDING, State.RESTARTING):
    w.clusters.ensure_cluster_is_running("0123-456789-abcde")
elif cluster.state == State.TERMINATED:
    w.clusters.start_and_wait(cluster_id="0123-456789-abcde")
elif cluster.state == State.ERROR:
    reason = cluster.termination_reason
    print(f"Cluster error: {reason.code} — {reason.parameters}")
    # Common: CLOUD_PROVIDER_LAUNCH_FAILURE, INSTANCE_POOL_CLUSTER_FAILURE

---

SPARK_DRIVER_OOM

java.lang.OutOfMemoryError: Java heap space
SparkException: Job aborted due to stage failure

Cause: Driver or executor running out of memory.