Implement Databricks API rate limiting, backoff, and idempotency patterns.
Use when handling rate limit errors, implementing retry logic,
or optimizing API request throughput for Databricks.
Trigger with phrases like "databricks rate limit", "databricks throttling",
"databricks 429", "databricks retry", "databricks backoff".
Implement Databricks API rate limiting, backoff, and idempotency patterns.
Use when handling rate limit errors, implementing retry logic,
or optimizing API request throughput for Databricks.
Trigger with phrases like "databricks rate limit", "databricks throttling",
"databricks 429", "databricks retry", "databricks backoff".
allowed-tools
Read, Write, Edit
version
1.0.0
license
MIT
author
Jeremy Longshore <jeremy@intentsolutions.io>
compatible-with
claude-code, codex, openclaw
tags
["saas","databricks","api"]
Databricks Rate Limits
Overview
Handle Databricks API rate limits with exponential backoff, token-bucket queuing, and idempotent job submissions. The API returns HTTP 429 with a Retry-After header when limits are exceeded. The SDK has built-in retries for transient errors, but custom logic is needed for bulk operations.
Prerequisites
databricks-sdk installed
Understanding of async patterns for batch operations
Step 3: Token-Bucket Rate Limiter for Bulk Operations
Prevent bursts when iterating over hundreds of resources.
import threading
import time
classRateLimiter:
"""Token-bucket rate limiter for Databricks API calls."""def__init__(self, requests_per_second: float = 8.0):
self._interval = 1.0 / requests_per_second
self._lock = threading.Lock()
self._last_request = 0.0defacquire(self):
"""Block until the next request slot is available."""withself._lock:
now = time.monotonic()
wait = self._last_request + self._interval - now
if wait > 0:
time.sleep(wait)
self._last_request = time.monotonic()
# Usage: enumerate jobs without hitting limits
limiter = RateLimiter(requests_per_second=8)
deflist_all_job_runs(w, job_ids: list[int]) -> dict:
results = {}
for job_id in job_ids:
limiter.acquire()
runs = list(w.jobs.list_runs(job_id=job_id, limit=5))
results[job_id] = runs
return results
Step 4: Concurrent Batch Processing with Throttle
from concurrent.futures import ThreadPoolExecutor, as_completed
defbatch_run_jobs(w, job_ids: list[int], max_concurrent: int = 5) -> dict:
"""Run multiple jobs with concurrency throttling."""
results = {}
defrun_one(job_id):
limiter.acquire()
try:
run = w.jobs.run_now(job_id=job_id)
return job_id, {"run_id": run.run_id, "status": "submitted"}
except TooManyRequests:
time.sleep(5)
run = w.jobs.run_now(job_id=job_id)
return job_id, {"run_id": run.run_id, "status": "submitted_after_retry"}
except ResourceConflict:
return job_id, {"status": "already_running"}
with ThreadPoolExecutor(max_workers=max_concurrent) as executor:
futures = {executor.submit(run_one, jid): jid for jid in job_ids}
for future in as_completed(futures):
job_id, result = future.result()
results[job_id] = result
return results
Step 5: Idempotent Job Submissions
Prevent duplicate runs when retrying failed submissions using idempotency_token.
import hashlib
from datetime import datetime
defsubmit_idempotent(w, job_id: int, params: dict | None = None) -> int:
"""Submit a job run with idempotency — safe to retry."""# Deterministic token: same job + date + params = same token
token_input = f"{job_id}-{datetime.utcnow().strftime('%Y-%m-%d')}-{sorted(params.items()) if params else''}"
idempotency_token = hashlib.sha256(token_input.encode()).hexdigest()[:32]
run = w.jobs.run_now(
job_id=job_id,
idempotency_token=idempotency_token,
notebook_params=params or {},
)
return run.run_id
# Calling twice with same inputs on the same day returns the same run_id
run1 = submit_idempotent(w, 456, params={"date": "2025-03-01"})
run2 = submit_idempotent(w, 456, params={"date": "2025-03-01"})
assert run1 == run2 # No duplicate run created
Output
Retry-safe API calls handling 429 and 503 with exponential backoff
Token-bucket rate limiter for bulk resource enumeration
Thread-pool batch runner with configurable concurrency
