databricks-notebooks

name	databricks-notebooks
description	Work with Databricks notebooks using the REST API - list, read, modify, and write notebooks back to Databricks workspace

Databricks Notebooks Management

Overview

This skill enables working with Databricks notebooks using the Databricks Workspace REST API. You can list, read, modify, and write notebooks without needing the databricks-cli package or any Python dependencies - just curl and the REST API.

When to Use

• Reading notebook source code from Databricks
• Modifying notebooks (adding filters, changing queries, refactoring)
• Uploading modified notebooks back to Databricks
• Listing notebooks in a workspace directory
• Working around databricks-cli compatibility issues with urllib3

Prerequisites

User must have ~/.databrickscfg configured with workspace credentials:

[DEFAULT]
host = https://workspace-url.cloud.databricks.net/
token = dapi...

Reading Credentials

Always read credentials from the config file:

DATABRICKS_HOST=$(grep "^host" ~/.databrickscfg | cut -d= -f2 | tr -d ' ')
DATABRICKS_TOKEN=$(grep "^token" ~/.databrickscfg | cut -d= -f2 | tr -d ' ')

Common Operations

1. List Notebooks in Directory

curl -s -X GET \
  "${DATABRICKS_HOST}/api/2.0/workspace/list" \
  -H "Authorization: Bearer ${DATABRICKS_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"path": "/Users/[e-mail]"}' | jq '.'

Returns objects with:

• object_type: NOTEBOOK, DIRECTORY, REPO, LIBRARY
• path: Full path to the object
• language: PYTHON, SQL, SCALA, R (for notebooks)

2. Export (Read) Notebook

Download notebook source code to local file:

curl -s -X GET \
  "${DATABRICKS_HOST}/api/2.0/workspace/export" \
  -H "Authorization: Bearer ${DATABRICKS_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"path": "/Users/[e-mail]/notebook_name", "format": "SOURCE"}' \
  | jq -r '.content' | base64 -d > /tmp/notebook.py

Important:

• Content is base64-encoded in the API response
• Use "format": "SOURCE" for .py format
• Use "format": "JUPYTER" for .ipynb format
• Only exports source code, NOT cell outputs

3. Import (Write) Notebook

Upload modified notebook back to Databricks:

content=$(base64 -w 0 /tmp/notebook.py)
curl -s -X POST \
  "${DATABRICKS_HOST}/api/2.0/workspace/import" \
  -H "Authorization: Bearer ${DATABRICKS_TOKEN}" \
  -H "Content-Type: application/json" \
  -d "{\"path\": \"/Users/[e-mail]/notebook_name\", \"content\": \"$content\", \"format\": \"SOURCE\", \"language\": \"PYTHON\", \"overwrite\": true}"

Important:

• Content must be base64-encoded
• Always use "overwrite": true to update existing notebooks
• Returns {"object_id": ...} on success

Typical Workflow

1. Read credentials from ~/.databrickscfg
2. List notebooks (if needed to find exact path)
3. Export notebook to /tmp/notebook.py
4. Read and modify the file using Read/Edit tools
5. Import notebook back to Databricks
6. Inform user to refresh Databricks UI

Notebook Format

Databricks notebooks use special comment markers:

# Databricks notebook source
# First cell code here

# COMMAND ----------

# Second cell code here

# COMMAND ----------

# Third cell code here

Each # COMMAND ---------- separates cells.

Key Limitations

• Cannot read cell outputs - API only returns source code
• Cannot see execution results - No access to dataframes, variables, or display() output
• Cannot execute cells - Cannot run code remotely
• No cluster attachment - Cannot interact with running clusters
• Manual refresh required - User must refresh Databricks UI to see changes

Common Use Cases

Modify Query to Limit Results

# Before:
df.filter(condition).display()

# After:
df.filter(condition).limit(10).display()

Add Timestamp Filter

# Before:
df = spark.table("table_name")

# After:
df = spark.table("table_name").filter(F.col("timestamp") > "2024-01-01")

Change Display Format

# Before:
display(df)

# After:
df.show(10, truncate=False)

Error Handling

• Missing credentials: Check ~/.databrickscfg exists
• Permission denied: Verify token has workspace access
• Path not found: Use list operation to find correct path
• Import fails: Ensure base64 encoding is correct

Best Practices

• Store notebooks in /tmp/ when working with them locally
• Always verify changes by exporting after import
• Use jq for pretty-printing JSON responses
• Include full paths starting with /Users/ or /Shared/
• Keep original formatting when modifying code