databricks-rate-limits

Implement Databricks API rate limiting, backoff, and idempotency patterns. Use when handling rate limit errors, implementing retry logic, or optimizing API request throughput for Databricks. Trigger with phrases like "databricks rate limit", "databricks throttling", "databricks 429", "databricks retry", "databricks backoff".

Ir para a instalação

Repositório: ComeOnOliver/skillshub
Última atividade na origem: 23 de março de 2026 às 17:43
Idioma detectado do SKILL.md: inglês
Estrelas: 58
Forks: 20

Opções de instalação

Por padrão, está selecionado o prompt que primeiro revisa a origem. Você pode mudar para um comando direto ou baixar uma cópia local.

Revise os arquivos de origem

Leia o SKILL.md e os arquivos complementares exibidos pelo SkillsMP antes de decidir se vai instalar.

Voltar à pesquisa Voltar à instalação

name	databricks-rate-limits
description	Implement Databricks API rate limiting, backoff, and idempotency patterns. Use when handling rate limit errors, implementing retry logic, or optimizing API request throughput for Databricks. Trigger with phrases like "databricks rate limit", "databricks throttling", "databricks 429", "databricks retry", "databricks backoff".
allowed-tools	Read, Write, Edit
version	1.0.0
license	MIT
author	Jeremy Longshore <jeremy@intentsolutions.io>
compatible-with	claude-code, codex, openclaw
tags	["saas","databricks","api"]

Databricks Rate Limits

Overview

Handle Databricks API rate limits with exponential backoff, token-bucket queuing, and idempotent job submissions. The API returns HTTP 429 with a Retry-After header when limits are exceeded. The SDK has built-in retries for transient errors, but custom logic is needed for bulk operations.

Prerequisites

databricks-sdk installed
Understanding of async patterns for batch operations

Instructions

Step 1: Understand Rate Limit Tiers

Databricks enforces per-endpoint, per-workspace rate limits.

API Category	Approx. Limit	Notes
Jobs API (create/run)	~10 req/sec	Per workspace
Jobs API (list/get)	~30 req/sec	Read endpoints more generous
Clusters API	~10 req/sec	Create/start are expensive
DBFS / Files API	~10 req/sec	Uploads have 1MB/5MB size limits
SQL Statement API	~10 concurrent	Concurrent execution limit
Unity Catalog	~100 req/min	Permission checks add up fast
Model Serving	Varies	ITPM/OTPM/QPH limits per endpoint

from databricks.sdk.errors import TooManyRequests, ResourceConflict

w = WorkspaceClient()
try:
    w.jobs.run_now(job_id=123)
except TooManyRequests as e:
    print(f"Rate limited. Retry after: {e.retry_after_secs}s")
except ResourceConflict as e:
    print(f"Conflict (409): {e.message}")  # Job already running

Step 2: Exponential Backoff with Jitter

import time
 random
 functools  wraps
 databricks.sdk.errors  TooManyRequests, TemporarilyUnavailable

 ():
    
     ():

         ():
             attempt  (max_retries):
                :
                     func(*args, **kwargs)
                 TooManyRequests  e:
                     attempt == max_retries - :
                        
                    delay = (base_delay * ( ** attempt), max_delay)
                    jitter = random.uniform(, delay * )
                    wait = e.retry_after_secs  (delay + jitter)
                    ()
                    time.sleep(wait)
                 TemporarilyUnavailable:
                     attempt == max_retries - :
                        
                    delay = (base_delay * ( ** attempt), max_delay)
                    ()
                    time.sleep(delay)
             func(*args, **kwargs)
         wrapper
     decorator


 ():
     w.jobs.get(job_id)

import threading
import time

class RateLimiter:
    """Token-bucket rate limiter for Databricks API calls."""

    def __init__(self, requests_per_second: float = 8.0):
        self._interval = 1.0 / requests_per_second
        self._lock = threading.Lock()
        self._last_request = 0.0

    def acquire(self):
        """Block until the next request slot is available."""
        with self._lock:
            now = time.monotonic()
            wait = self._last_request + self._interval - now
            if wait > 0:
                time.sleep(wait)
            self._last_request = time.monotonic()

# Usage: enumerate jobs without hitting limits
limiter = RateLimiter(requests_per_second=8)

def list_all_job_runs(w, job_ids: list[int]) -> dict:
    results = {}
    for job_id in job_ids:
        limiter.acquire()
        runs = list(w.jobs.list_runs(job_id=job_id, limit=5))
        results[job_id] = runs
    return results

from concurrent.futures import ThreadPoolExecutor, as_completed

def batch_run_jobs(w, job_ids: list[int], max_concurrent: int = 5) -> dict:
    """Run multiple jobs with concurrency throttling."""
    results = {}

    def run_one(job_id):
        limiter.acquire()
        try:
            run = w.jobs.run_now(job_id=job_id)
            return job_id, {"run_id": run.run_id, "status": "submitted"}
        except TooManyRequests:
            time.sleep(5)
            run = w.jobs.run_now(job_id=job_id)
            return job_id, {"run_id": run.run_id, "status": "submitted_after_retry"}
        except ResourceConflict:
            return job_id, {"status": "already_running"}

    with ThreadPoolExecutor(max_workers=max_concurrent) as executor:
        futures = {executor.submit(run_one, jid): jid for jid in job_ids}
        for future in as_completed(futures):
            job_id, result = future.result()
            results[job_id] = result

    return results

import hashlib
from datetime import datetime

def submit_idempotent(w, job_id: int, params: dict | None = None) -> int:
    """Submit a job run with idempotency — safe to retry."""
    # Deterministic token: same job + date + params = same token
    token_input = f"{job_id}-{datetime.utcnow().strftime('%Y-%m-%d')}-{sorted(params.items()) if params else ''}"
    idempotency_token = hashlib.sha256(token_input.encode()).hexdigest()[:32]

    run = w.jobs.run_now(
        job_id=job_id,
        idempotency_token=idempotency_token,
        notebook_params=params or {},
    )
    return run.run_id

# Calling twice with same inputs on the same day returns the same run_id
run1 = submit_idempotent(w, 456, params={"date": "2025-03-01"})
run2 = submit_idempotent(w, 456, params={"date": "2025-03-01"})
assert run1 == run2  # No duplicate run created

Error	HTTP	Solution
`TooManyRequests`	429	Use `Retry-After` header, fall back to exponential backoff
`TemporarilyUnavailable`	503	Retry with 5-10s delay; check status.databricks.com
`ResourceConflict`	409	Job already running — check `list_runs()` before submitting
`TimeoutError`	-	Increase SDK timeout: `WorkspaceClient(timeout=120)`
Sustained rate limiting	429	Reduce concurrency, spread load across time windows

import requests

resp = requests.get(
    f"{w.config.host}/api/2.1/jobs/list",
    headers={"Authorization": f"Bearer {w.config.token}"},
)
print(f"Status: {resp.status_code}")
print(f"Retry-After: {resp.headers.get('Retry-After', 'N/A')}")

limiter = RateLimiter(requests_per_second=5)
terminated = 0
for cluster in w.clusters.list():
    if cluster.state.value == "TERMINATED" and cluster.cluster_name.startswith("dev-"):
        limiter.acquire()
        w.clusters.permanent_delete(cluster_id=cluster.cluster_id)
        terminated += 1
print(f"Cleaned up {terminated} dev clusters")