databricks-isv-u2m

<!-- skill-version: 1.0.0 --> # U2M (User-to-Machine) Authentication (ISV) Use this skill when implementing or testing **U2M** (browser or pre-obtained token) authentication for Databricks partner integrations. ## Important: M2M client_id ≠ U2M OAuth app - The **service principal** client_id used for **OAuth M2M** (`DATABRICKS_CLIENT_ID`) is **not** registered for the authorization code (browser) flow. Using it with the authorize URL or with SDK external-browser causes: "OAuth application with client_id: '...' not available in Databricks account". - For **browser-based U2M** use either: (1) SDK external-browser with **no** client_id (Databricks built-in app), or (2) a **separate** custom OAuth app (created in the account) with a redirect URI. - We use **separate env var names** for U2M custom apps: `DATABRICKS_U2M_CLIENT_ID` and `DATABRICKS_U2M_CLIENT_SECRET` to avoid conflicts with M2M credentials (`DATABRICKS_CLIENT_ID` / `DATABRICKS_CLIENT_SECRET`) in the same env file. ## Three Flows | Flow | When to Use | Required Variables | |------|-------------|---------------------| | **external-browser** | No custom OAuth app; SDK opens browser | `DATABRICKS_HOST` only | | **custom-oauth-app** | Your registered OAuth app + configurable redirect URI | `DATABRICKS_HOST`, `DATABRICKS_U2M_CLIENT_ID`; optional redirect_uri, client_secret | | **token-env** | Pre-obtained token (hosted callback, refresh, CLI) | `DATABRICKS_HOST`, `DATABRICKS_ACCESS_TOKEN` or `DATABRICKS_TOKEN` | --- ## Flow 1: External-Browser (SDK Built-in App) Use the SDK's built-in OAuth app (`databricks-cli`). Do **not** set or pass `DATABRICKS_CLIENT_ID` / `DATABRICKS_CLIENT_SECRET`. ### Python SDK ```python from databricks.sdk import WorkspaceClient from databricks.sdk.config import Config config = Config( host="https://myworkspace.cloud.databricks.com", auth_type="external-browser" ) # Unset M2M credentials before this if they exist in environment client = WorkspaceClient(config=config) # Get token for other uses (e.g., REST API, JDBC) token = config.authenticate()["Authorization"].replace("Bearer ", "") ``` ### Java SDK ```java import com.databricks.sdk.core.DatabricksConfig; import java.util.Arrays; import java.util.Map; public static String getTokenViaExternalBrowser(String host) { String normalizedHost = host.startsWith("https://") ? host : "https://" + host; DatabricksConfig config = new DatabricksConfig() .setHost(normalizedHost) .setAuthType("external-browser") .setScopes(Arrays.asList("all-apis")) // REQUIRED .setOAuthRedirectUrl("http://localhost:8020"); // REQUIRED for built-in app config.resolve(); // REQUIRED: initializes HTTP client Map<String, String> headers = config.authenticate(); return headers.get("Authorization").substring("Bearer ".length()); } ``` ### Token Caching The SDK caches tokens. To force a fresh login: - **Python SDK:** Delete `~/.databricks/token-cache.json` - **Java SDK:** Delete `~/.config/databricks-sdk-java/oauth/` - Also clear browser cookies for the workspace and SSO provider ### Redirect URI for Built-in App | SDK | Default | Registered for `databricks-cli` | Action | |-----|---------|--------------------------------|--------| | Python SDK | `http://localhost:8020` | `http://localhost:8020` | None needed | | Java SDK v0.54.0 | `http://localhost:8080/callback` | `http://localhost:8020` | **Must set explicitly** | --- ## Flow 2: Custom-OAuth-App (Authorization Code + PKCE) Register a custom OAuth app in Databricks and implement PKCE flow. ### Custom OAuth App Setup (Prerequisite) 1. Log in to Databricks account console (`https://accounts.cloud.databricks.com/`) 2. Go to **Settings → App connections → Add connection** 3. Create an app. Note the **Client ID**. If confidential, generate a **Client Secret**. 4. Add **Redirect URIs** — e.g., `http://localhost:8080/callback` > **Three-way consistency rule:** The redirect URI must exactly match in three places: (1) the skill/code default, (2) the connector code, and (3) the **App connections → your app → Redirect URIs** in Databricks. A mismatch causes a silent OAuth callback failure with no clear error. The port `8080` is the default — if you use a different port, update all three locations. ### PKCE Flow Implementation **Authorize URL:** ``` {host}/oidc/v1/authorize ?response_type=code &client_id={client_id} &redirect_uri={redirect_uri} &scope=all-apis &code_challenge={S256_challenge} &code_challenge_method=S256 &state={state} ``` **Token URL:** ``` POST {host}/oidc/v1/token Content-Type: application/x-www-form-urlencoded grant_type=authorization_code &code={code_from_redirect} &redirect_uri={same_as_authorize} &client_id={client_id} &code_verifier={verifier} ``` ### Public vs Confidential OAuth Apps Databricks OAuth apps registered in App connections can be **public** (no client secret) or **confidential** (registered with a client secret). | App type | `client_secret` in token exchange | Error if missing | |----------|-----------------------------------|-----------------| | **Public** | Omit `client_secret` | n/a | | **Confidential** | **Required** — must include `client_secret` in POST body | `{"error": "invalid_client"}` | > ⚠️ **DIAGNOSTIC: `invalid_client` on token exchange** > Cause: The OAuth app is **confidential** (has a client secret registered) but `client_secret` was omitted from the token exchange POST body. > Fix: Add `client_secret=<secret>` to the POST body. The secret is shown once at app creation in Databricks App connections — if lost, regenerate it. > > Full token exchange body for confidential apps: > ``` > grant_type=authorization_code > &code=<code> > &redirect_uri=<uri> > &client_id=<id> > &code_verifier=<verifier> > &client_secret=<secret> ← required for confidential apps > ``` ### Python PKCE Implementation ```python import secrets import hashlib import base64 import webbrowser import threading from http.server import HTTPServer, BaseHTTPRequestHandler from urllib.parse import urlencode, urlparse, parse_qs import requests def generate_pkce(): """Generate PKCE code_verifier and code_challenge.""" verifier = secrets.token_urlsafe(64) digest = hashlib.sha256(verifier.encode()).digest() challenge = base64.urlsafe_b64encode(digest).rstrip(b"=").decode() return verifier, challenge class CallbackHandler(BaseHTTPRequestHandler): auth_code = None callback_received = threading.Event() def do_GET(self): if "/callback" in self.path: query = parse_qs(urlparse(self.path).query) CallbackHandler.auth_code = query.get("code", [None])[0] self.send_response(200) self.end_headers() self.wfile.write(b"Authentication complete. Close this tab.") CallbackHandler.callback_received.set() else: self.send_response(204) self.end_headers() def log_message(self, format, *args): pass def run_pkce_flow(host, client_id, client_secret=None, redirect_uri="http://localhost:8080/callback", timeout=120): """Complete PKCE flow and return access token.""" # Parse port from redirect URI port = int(urlparse(redirect_uri).port or 8080) # Generate PKCE values verifier, challenge = generate_pkce() state = secrets.token_urlsafe(16) # Build authorize URL host = host.rstrip("/") if not host.startswith("https://"): host = "https://" + host auth_params = { "response_type": "code", "client_id": client_id, "redirect_uri": redirect_uri, "scope": "all-apis", "code_challenge": challenge, "code_challenge_method": "S256", "state": state, } auth_url = f"{host}/oidc/v1/authorize?{urlencode(auth_params)}" # Start callback server CallbackHandler.auth_code = None CallbackHandler.callback_received.clear() server = HTTPServer(("localhost", port), CallbackHandler) server.timeout = timeout # CRITICAL: prevents infinite blocking # Open browser webbrowser.open(auth_url) print(f"Opened browser for authentication...") # Wait for callback while not CallbackHandler.callback_received.is_set(): server.handle_request() server.server_close() # Use server_close(), NOT shutdown() if not CallbackHandler.auth_code: raise RuntimeError("No auth code received") # Exchange code for token token_data = { "grant_type": "authorization_code", "code": CallbackHandler.auth_code, "redirect_uri": redirect_uri, "client_id": client_id, "code_verifier": verifier, } if client_secret: token_data["client_secret"] = client_secret token_url = f"{host}/oidc/v1/token" resp = requests.post(token_url, data=token_data) resp.raise_for_status() return resp.json()["access_token"] ``` ### Java SDK with Custom OAuth App ```java import com.databricks.sdk.core.DatabricksConfig; import java.util.Arrays; import java.util.Map; public static String getTokenViaCustomOAuthApp( String host, String clientId, String clientSecret, String redirectUri) { String normalizedHost = host.startsWith("https://") ? host : "https://" + host; DatabricksConfig config = new DatabricksConfig() .setHost(normalizedHost) .setAuthType("external-browser") .setClientId(clientId) .setScopes(Arrays.asList("all-apis")); if (clientSecret != null && !clientSecret.isBlank()) { config.setClientSecret(clientSecret); } if (redirectUri != null && !redirectUri.isBlank()) { config.setOAuthRedirectUrl(redirectUri); } config.resolve(); Map<String, String> headers = config.authenticate(); return headers.get("Authorization").substring("Bearer ".length()); } ``` ### Go PKCE Implementation ```go package main import ( "crypto/rand" "crypto/sha256" "encoding/base64" "fmt" "net/http" "net/url" "strings" ) func generatePKCE() (verifier, challenge string) { b := make([]byte, 64) rand.Read(b) verifier = base64.RawURLEncoding.EncodeToString(b) h := sha256.Sum256([]byte(verifier)) challenge = base64.RawURLEncoding.EncodeToString(h[:]) return } func runPKCEFlow(host, clientID, clientSecret, redirectURI string, timeout int) (string, error) { verifier, challenge := generatePKCE() state := generateRandomString(16) // Parse redirect URI for port u, _ := url.Parse(redirectURI) port := u.Port() if port == "" { port = "8080" } // Build authorize URL if !strings.HasPrefix(host, "https://") { host = "https://" + host } authParams := url.Values{ "response_type": {"code"}, "client_id": {clientID}, "redirect_uri": {redirectURI}, "scope": {"all-apis"},