databricks-isv-python-sdk

<!-- skill-version: 1.0.0 --> # Databricks SDK for Python (ISV) Use this skill when implementing or testing **Databricks SDK for Python** (`databricks-sdk`) integrations for PWAF-compliant workspace management, Unity Catalog, Jobs, and REST-style API access. ## PWAF Documentation Links | Resource | URL | |----------|-----| | **PWAF Home** | https://databrickslabs.github.io/partner-architecture/ | | **Authentication Best Practices** | https://databrickslabs.github.io/partner-architecture/isv-partners/lakehouse-patterns/access-auth/ | | **OAuth M2M Guide** | https://databrickslabs.github.io/partner-architecture/isv-partners/lakehouse-patterns/access-auth/oauth-m2m | | **OAuth U2M Guide** | https://databrickslabs.github.io/partner-architecture/isv-partners/lakehouse-patterns/access-auth/oauth-u2m | | **User-Agent Telemetry** | https://databrickslabs.github.io/partner-architecture/isv-partners/telemetry-attribution/sdks | | **Python SDK Docs** | https://docs.databricks.com/aws/en/dev-tools/sdk-python | | **Python SDK GitHub** | https://github.com/databricks/databricks-sdk-py | ## Requirements - **SDK:** `databricks-sdk` 0.20.0+ - **Python:** 3.8+ - **Install:** `pip install databricks-sdk` ## Authentication Decision Guide ``` Which authentication method to use? Production / automated workloads? → OAuth M2M (client credentials) ✅ RECOMMENDED User-interactive flows? → U2M Custom OAuth App (PKCE) ✅ SUPPORTED Already have an OAuth access token? → U2M Token-Env ✅ SUPPORTED Local development/testing only? → PAT (Personal Access Token) ⚠️ LIMITED ``` ## Authentication Comparison | Method | PWAF Status | Auto Token Refresh | Browser Required | Use Case | |--------|-------------|-------------------|------------------|----------| | PAT | ⚠️ Limited | No | No | Testing only | | OAuth M2M | ✅ Recommended | Yes (SDK handles) | No | Production/automated | | U2M Custom OAuth App | ✅ Supported | No | Yes | User-interactive | | U2M Token-Env | ✅ Supported | No | No | Headless/CI | --- ## CLIENT_ID Distinction (CRITICAL) | Variable | Purpose | Value Source | Use Case | |----------|---------|--------------|----------| | `DATABRICKS_CLIENT_ID` | M2M Service Principal | Workspace → Identity and access → Service principals → UUID | Automated/production | | `DATABRICKS_U2M_CLIENT_ID` | U2M Custom OAuth App | Account Console → Settings → App connections → Client ID | User-interactive | **Never mix these.** M2M uses service principal credentials for server-to-server auth. U2M custom OAuth uses a registered app for user login. --- ## Token Lifetime Summary | Token Type | Typical TTL | Refresh Mechanism | |------------|-------------|-------------------| | PAT | 90 days (configurable) | Manual regeneration | | M2M access_token | 1 hour | SDK auto-refreshes | | U2M access_token | 1 hour | Manual re-auth or refresh_token | --- ## User-Agent (Required per PWAF) Call once before creating `WorkspaceClient`: ```python from databricks.sdk import useragent useragent.with_partner("YourCompany") useragent.with_product("YourCompany_YourProduct", "1.0.0") ``` **Alternative (on WorkspaceClient):** ```python from databricks.sdk import WorkspaceClient client = WorkspaceClient( host="https://myworkspace.cloud.databricks.com", token="dapi...", product="YourCompany_YourProduct", product_version="1.0.0" ) ``` --- ## Environment Variables Reference | Variable | Required For | Description | |----------|-------------|-------------| | `DATABRICKS_HOST` | All | Workspace URL (e.g., `https://myworkspace.cloud.databricks.com`) | | `DATABRICKS_TOKEN` | PAT | Personal access token | | `DATABRICKS_CLIENT_ID` | OAuth M2M | Service principal UUID | | `DATABRICKS_CLIENT_SECRET` | OAuth M2M | Service principal OAuth secret | | `DATABRICKS_U2M_CLIENT_ID` | U2M Custom OAuth | Custom OAuth app client ID from App connections | | `DATABRICKS_U2M_CLIENT_SECRET` | U2M Custom OAuth (optional) | Custom OAuth app client secret (if confidential) | | `DATABRICKS_REDIRECT_URI` | U2M Custom OAuth (optional) | Custom redirect URI (default: `http://localhost:8080/callback`) | | `DATABRICKS_ACCESS_TOKEN` | U2M Token-Env | Pre-obtained OAuth access token | | `APP_AUTH_TYPE` | Multi-auth | App-level auth selector: `pat`, `oauth_m2m`, `oauth_u2m`, `u2m_token_env` | **Important:** - Use `APP_AUTH_TYPE` (not `DATABRICKS_AUTH_TYPE`) for app-level auth selection because the SDK reads `DATABRICKS_AUTH_TYPE` internally. - Do not mix M2M and U2M environment variables. Use `env -i` for clean test environments. --- ## Config Parameters Reference | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `host` | str | Yes | Workspace URL (include `https://`) | | `token` | str | PAT only | Personal access token | | `client_id` | str | OAuth M2M | Service principal UUID | | `client_secret` | str | OAuth M2M | Service principal OAuth secret | | `auth_type` | str | OAuth M2M | Set to `"oauth-m2m"` for M2M (avoids default credential resolution) | | `azure_use_msi` | bool | Azure only | Enable Azure Managed Identity | | `azure_client_id` | str | Azure MSI | Client ID for user-assigned MSI | | `product` | str | Telemetry | Product name for User-Agent | | `product_version` | str | Telemetry | Product version for User-Agent | **Critical:** When using OAuth M2M, always pass `auth_type="oauth-m2m"` to Config to avoid "cannot configure default credentials" error. --- ## Complete Examples ### PAT Authentication (Testing Only) ```python import os from databricks.sdk import WorkspaceClient, useragent useragent.with_partner("YourCompany") useragent.with_product("YourProduct", "1.0.0") def get_client_pat(): host = os.environ.get("DATABRICKS_HOST", "").strip() if not host.startswith("https://"): host = f"https://{host}" token = os.environ.get("DATABRICKS_TOKEN", "").strip() if not token: raise ValueError("DATABRICKS_TOKEN required") return WorkspaceClient(host=host, token=token) client = get_client_pat() me = client.current_user.me() print(f"Authenticated as: {me.user_name}") ``` ### OAuth M2M (Recommended for Production) ```python import os from databricks.sdk import WorkspaceClient, useragent from databricks.sdk.config import Config useragent.with_partner("YourCompany") useragent.with_product("YourProduct", "1.0.0") def get_client_m2m(): host = os.environ.get("DATABRICKS_HOST", "").strip() if not host.startswith("https://"): host = f"https://{host}" config = Config( host=host, client_id=os.environ.get("DATABRICKS_CLIENT_ID", ""), client_secret=os.environ.get("DATABRICKS_CLIENT_SECRET", ""), auth_type="oauth-m2m", # CRITICAL: Required for M2M ) return WorkspaceClient(config=config) client = get_client_m2m() # SDK automatically handles token refresh tables = list(client.tables.list("samples", "nyctaxi")) print(f"Found {len(tables)} tables") ``` ### OAuth U2M Custom OAuth App (PKCE) ```python import os import base64 import hashlib import secrets import string import webbrowser from http.server import HTTPServer, BaseHTTPRequestHandler from threading import Thread from urllib.parse import parse_qs, urlparse, urlencode import requests from databricks.sdk import WorkspaceClient DEFAULT_REDIRECT_URI = "http://localhost:8080/callback" def pkce_verifier_and_challenge(): """Generate PKCE code_verifier and code_challenge per RFC 7636.""" allowed = string.ascii_letters + string.digits + "-._~" code_verifier = "".join(secrets.choice(allowed) for _ in range(64)) digest = hashlib.sha256(code_verifier.encode()).digest() code_challenge = base64.urlsafe_b64encode(digest).decode().rstrip("=") return code_verifier, code_challenge def run_u2m_flow(host: str, client_id: str, redirect_uri: str = None, client_secret: str = None, scope: str = "all-apis") -> dict: """Run OAuth 2.0 authorization code flow with PKCE.""" redirect_uri = redirect_uri or DEFAULT_REDIRECT_URI if not host.startswith("https://"): host = f"https://{host}" code_verifier, code_challenge = pkce_verifier_and_challenge() state = secrets.token_urlsafe(32) auth_url = f"{host}/oidc/v1/authorize?" + urlencode({ "response_type": "code", "client_id": client_id, "redirect_uri": redirect_uri, "scope": scope, "code_challenge": code_challenge, "code_challenge_method": "S256", "state": state, }) code_holder = [] parsed = urlparse(redirect_uri) port = parsed.port or 8080 class Handler(BaseHTTPRequestHandler): def do_GET(self): q = parse_qs(urlparse(self.path).query) code_holder.append(q.get("code", [None])[0]) self.send_response(200) self.send_header("Content-type", "text/html; charset=utf-8") self.end_headers() self.wfile.write(b"<html><body><p>Authentication successful.</p></body></html>") def log_message(self, *args): pass server = HTTPServer(("localhost", port), Handler) Thread(target=server.handle_request, daemon=True).start() webbrowser.open(auth_url) print(f"Waiting for redirect at {redirect_uri}...") server.handle_request() if not code_holder[0]: raise RuntimeError("No authorization code received") token_data = { "grant_type": "authorization_code", "code": code_holder[0], "redirect_uri": redirect_uri, "client_id": client_id, "code_verifier": code_verifier, } if client_secret: token_data["client_secret"] = client_secret resp = requests.post(f"{host}/oidc/v1/token", data=token_data) resp.raise_for_status() return resp.json() def get_client_u2m(): host = os.environ.get("DATABRICKS_HOST", "").strip() client_id = os.environ.get("DATABRICKS_U2M_CLIENT_ID", "").strip() redirect_uri = os.environ.get("DATABRICKS_REDIRECT_URI", "") or DEFAULT_REDIRECT_URI client_secret = os.environ.get("DATABRICKS_U2M_CLIENT_SECRET", "").strip() or None tokens = run_u2m_flow(host, client_id, redirect_uri, client_secret)