databricks-isv-u2m

name	databricks-isv-u2m
description	U2M (user-to-machine) OAuth flows for Databricks ISV integrations: external-browser, custom-oauth-app, token-env. Covers Python, Java, Go, and Node.js implementations. Use when implementing or debugging browser-based or pre-obtained token auth.

U2M (User-to-Machine) Authentication (ISV)

Use this skill when implementing or testing U2M (browser or pre-obtained token) authentication for Databricks partner integrations.

Important: M2M client_id ≠ U2M OAuth app

• The service principal client_id used for OAuth M2M (DATABRICKS_CLIENT_ID) is not registered for the authorization code (browser) flow. Using it with the authorize URL or with SDK external-browser causes: "OAuth application with client_id: '...' not available in Databricks account".
• For browser-based U2M use either: (1) SDK external-browser with no client_id (Databricks built-in app), or (2) a separate custom OAuth app (created in the account) with a redirect URI.
• We use separate env var names for U2M custom apps: DATABRICKS_U2M_CLIENT_ID and DATABRICKS_U2M_CLIENT_SECRET to avoid conflicts with M2M credentials (DATABRICKS_CLIENT_ID / DATABRICKS_CLIENT_SECRET) in the same env file.

Three Flows

Flow	When to Use	Required Variables
external-browser	No custom OAuth app; SDK opens browser	DATABRICKS_HOST only
custom-oauth-app	Your registered OAuth app + configurable redirect URI	DATABRICKS_HOST, DATABRICKS_U2M_CLIENT_ID; optional redirect_uri, client_secret
token-env	Pre-obtained token (hosted callback, refresh, CLI)	DATABRICKS_HOST, DATABRICKS_ACCESS_TOKEN or DATABRICKS_TOKEN

---

Flow 1: External-Browser (SDK Built-in App)

Use the SDK's built-in OAuth app (databricks-cli). Do not set or pass DATABRICKS_CLIENT_ID / DATABRICKS_CLIENT_SECRET.

Python SDK

from databricks.sdk import WorkspaceClient
from databricks.sdk.config import Config

config = Config(
    host="https://myworkspace.cloud.databricks.com",
    auth_type="external-browser"
)
# Unset M2M credentials before this if they exist in environment
client = WorkspaceClient(config=config)

# Get token for other uses (e.g., REST API, JDBC)
token = config.authenticate()["Authorization"].replace("Bearer ", "")

Java SDK

import com.databricks.sdk.core.DatabricksConfig;
import java.util.Arrays;
import java.util.Map;

public static String getTokenViaExternalBrowser(String host) {
    String normalizedHost = host.startsWith("https://") ? host : "https://" + host;

    DatabricksConfig config = new DatabricksConfig()
            .setHost(normalizedHost)
            .setAuthType("external-browser")
            .setScopes(Arrays.asList("all-apis"))        // REQUIRED
            .setOAuthRedirectUrl("http://localhost:8020"); // REQUIRED for built-in app

    config.resolve();  // REQUIRED: initializes HTTP client
    Map<String, String> headers = config.authenticate();

    return headers.get("Authorization").substring("Bearer ".length());
}

Token Caching

The SDK caches tokens. To force a fresh login:

• Python SDK: Delete ~/.databricks/token-cache.json
• Java SDK: Delete ~/.config/databricks-sdk-java/oauth/
• Also clear browser cookies for the workspace and SSO provider

Redirect URI for Built-in App

SDK	Default	Registered for databricks-cli	Action
Python SDK	http://localhost:8020	http://localhost:8020	None needed
Java SDK v0.54.0	http://localhost:8080/callback	http://localhost:8020	Must set explicitly

---

Flow 2: Custom-OAuth-App (Authorization Code + PKCE)

Register a custom OAuth app in Databricks and implement PKCE flow.

Custom OAuth App Setup (Prerequisite)

1. Log in to Databricks account console (https://accounts.cloud.databricks.com/)
2. Go to Settings → App connections → Add connection
3. Create an app. Note the Client ID. If confidential, generate a Client Secret.
4. Add Redirect URIs — e.g., http://localhost:8080/callback

Three-way consistency rule: The redirect URI must exactly match in three places: (1) the skill/code default, (2) the connector code, and (3) the App connections → your app → Redirect URIs in Databricks. A mismatch causes a silent OAuth callback failure with no clear error. The port 8080 is the default — if you use a different port, update all three locations.

PKCE Flow Implementation

Authorize URL:

{host}/oidc/v1/authorize
  ?response_type=code
  &client_id={client_id}
  &redirect_uri={redirect_uri}
  &scope=all-apis
  &code_challenge={S256_challenge}
  &code_challenge_method=S256
  &state={state}

Token URL:

POST {host}/oidc/v1/token
Content-Type: application/x-www-form-urlencoded

grant_type=authorization_code
&code={code_from_redirect}
&redirect_uri={same_as_authorize}
&client_id={client_id}
&code_verifier={verifier}

Public vs Confidential OAuth Apps

Databricks OAuth apps registered in App connections can be public (no client secret) or confidential (registered with a client secret).

App type	client_secret in token exchange	Error if missing
Public	Omit client_secret	n/a
Confidential	Required — must include client_secret in POST body	{"error": "invalid_client"}

⚠️ DIAGNOSTIC: invalid_client on token exchange Cause: The OAuth app is confidential (has a client secret registered) but client_secret was omitted from the token exchange POST body. Fix: Add client_secret=<secret> to the POST body. The secret is shown once at app creation in Databricks App connections — if lost, regenerate it.

Full token exchange body for confidential apps:

grant_type=authorization_code
&code=<code>
&redirect_uri=<uri>
&client_id=<id>
&code_verifier=<verifier>
&client_secret=<secret>   ← required for confidential apps

Python PKCE Implementation

import secrets
import hashlib
import base64
import webbrowser
import threading
from http.server import HTTPServer, BaseHTTPRequestHandler
from urllib.parse import urlencode, urlparse, parse_qs
import requests

def generate_pkce():
    """Generate PKCE code_verifier and code_challenge."""
    verifier = secrets.token_urlsafe(64)
    digest = hashlib.sha256(verifier.encode()).digest()
    challenge = base64.urlsafe_b64encode(digest).rstrip(b"=").decode()
    return verifier, challenge

class CallbackHandler(BaseHTTPRequestHandler):
    auth_code = None
    callback_received = threading.Event()
    
    def do_GET(self):
        if "/callback" in self.path:
            query = parse_qs(urlparse(self.path).query)
            CallbackHandler.auth_code = query.get("code", [None])[0]
            self.send_response(200)
            self.end_headers()
            self.wfile.write(b"Authentication complete. Close this tab.")
            CallbackHandler.callback_received.set()
        else:
            self.send_response(204)
            self.end_headers()
    
    def log_message(self, format, *args):
        pass

def run_pkce_flow(host, client_id, client_secret=None,
                  redirect_uri="http://localhost:8080/callback", timeout=120):
    """Complete PKCE flow and return access token."""
    
    # Parse port from redirect URI
    port = int(urlparse(redirect_uri).port or 8080)
    
    # Generate PKCE values
    verifier, challenge = generate_pkce()
    state = secrets.token_urlsafe(16)
    
    # Build authorize URL
    host = host.rstrip("/")
    if not host.startswith("https://"):
        host = "https://" + host
    
    auth_params = {
        "response_type": "code",
        "client_id": client_id,
        "redirect_uri": redirect_uri,
        "scope": "all-apis",
        "code_challenge": challenge,
        "code_challenge_method": "S256",
        "state": state,
    }
    auth_url = f"{host}/oidc/v1/authorize?{urlencode(auth_params)}"
    
    # Start callback server
    CallbackHandler.auth_code = None
    CallbackHandler.callback_received.clear()
    
    server = HTTPServer(("localhost", port), CallbackHandler)
    server.timeout = timeout  # CRITICAL: prevents infinite blocking
    
    # Open browser
    webbrowser.open(auth_url)
    print(f"Opened browser for authentication...")
    
    # Wait for callback
    while not CallbackHandler.callback_received.is_set():
        server.handle_request()
    
    server.server_close()  # Use server_close(), NOT shutdown()
    
    if not CallbackHandler.auth_code:
        raise RuntimeError("No auth code received")
    
    # Exchange code for token
    token_data = {
        "grant_type": "authorization_code",
        "code": CallbackHandler.auth_code,
        "redirect_uri": redirect_uri,
        "client_id": client_id,
        "code_verifier": verifier,
    }
    if client_secret:
        token_data["client_secret"] = client_secret
    
    token_url = f"{host}/oidc/v1/token"
    resp = requests.post(token_url, data=token_data)
    resp.raise_for_status()
    
    return resp.json()["access_token"]

Java SDK with Custom OAuth App

import com.databricks.sdk.core.DatabricksConfig;
import java.util.Arrays;
import java.util.Map;

public static String getTokenViaCustomOAuthApp(
        String host, String clientId, String clientSecret, String redirectUri) {
    
    String normalizedHost = host.startsWith("https://") ? host : "https://" + host;

    DatabricksConfig config = new DatabricksConfig()
            .setHost(normalizedHost)
            .setAuthType("external-browser")
            .setClientId(clientId)
            .setScopes(Arrays.asList("all-apis"));

    if (clientSecret != null && !clientSecret.isBlank()) {
        config.setClientSecret(clientSecret);
    }
    if (redirectUri != null && !redirectUri.isBlank()) {
        config.setOAuthRedirectUrl(redirectUri);
    }

    config.resolve();
    Map<String, String> headers = config.authenticate();
    return headers.get("Authorization").substring("Bearer ".length());
}

Go PKCE Implementation

package main

import (
    "crypto/rand"
    "crypto/sha256"
    "encoding/base64"
    "fmt"
    "net/http"
    "net/url"
    "strings"
)

func generatePKCE() (verifier, challenge string) {
    b := make([]byte, 64)
    rand.Read(b)
    verifier = base64.RawURLEncoding.EncodeToString(b)
    h := sha256.Sum256([]byte(verifier))
    challenge = base64.RawURLEncoding.EncodeToString(h[:])
    return
}

func runPKCEFlow(host, clientID, clientSecret, redirectURI string, timeout int) (string, error) {
    verifier, challenge := generatePKCE()
    state := generateRandomString(16)
    
    // Parse redirect URI for port
    u, _ := url.Parse(redirectURI)
    port := u.Port()
    if port == "" {
        port = "8080"
    }
    
    // Build authorize URL
    if !strings.HasPrefix(host, "https://") {
        host = "https://" + host
    }
    
    authParams := url.Values{
        "response_type":         {"code"},
        "client_id":            {clientID},
        "redirect_uri":         {redirectURI},
        "scope":                {"all-apis"},
        "code_challenge":       {challenge},
        "code_challenge_method": {"S256"},
        "state":                {state},
    }
    authURL := fmt.Sprintf("%s/oidc/v1/authorize?%s", host, authParams.Encode())
    
    // Start callback server and open browser
    codeChan := make(chan string, 1)
    go startCallbackServer(port, codeChan)
    openBrowser(authURL)
    
    // Wait for code
    code := <-codeChan
    
    // Exchange code for token
    return exchangeCodeForToken(host, clientID, clientSecret, code, redirectURI, verifier)
}

HTTPServer Pitfalls (Critical)

When implementing the local callback server:

Issue	Solution
Server blocks forever	Set server.timeout = 120 before handle_request()
shutdown() hangs	Use server.server_close() instead of server.shutdown()
Browser sends favicon requests	Loop on handle_request() until callback received
Port already in use	Kill previous process: lsof -ti:8080 | xargs kill -9

---

Flow 3: Token-Env (Pre-obtained Token)

Use this flow when the connector is not responsible for the browser interaction — the token was acquired elsewhere and injected at runtime. Common scenarios:

• ISV platform with server-side OAuth — the ISV's backend hosts the redirect URI (e.g., https://myapp.example.com/oauth/callback) instead of localhost. This is the same PKCE/authorization-code flow as custom-oauth-app, but the backend captures the auth code and exchanges it for a token rather than the connector itself. The connector receives only the finished access token.
• Refresh-token architecture — a long-lived refresh token is stored encrypted in the ISV's platform; the platform exchanges it for a short-lived access token and injects it as an env var before the connector runs.
• CLI pre-login — a user ran databricks auth login or the ISV's own login command beforehand; the resulting token is exported as DATABRICKS_ACCESS_TOKEN.
• CI/CD pipelines requiring user context — pipelines that must run as a real user (not a service principal) for audit/compliance; a token is minted at setup time and injected non-interactively.

If the connector is always the entry point for the user's browser interaction, use custom-oauth-app (PKCE built into the connector) instead.

Python

import os

token = os.environ.get("DATABRICKS_ACCESS_TOKEN") or os.environ.get("DATABRICKS_TOKEN")
host = os.environ["DATABRICKS_HOST"]

# For REST API
headers = {
    "Authorization": f"Bearer {token}",
    "User-Agent": "YourCompany_Product/1.0.0",
}

# For databricks-sdk
from databricks.sdk import WorkspaceClient
from databricks.sdk.config import Config

config = Config(host=host, token=token)
client = WorkspaceClient(config=config)

Java JDBC

String accessToken = System.getenv("DATABRICKS_ACCESS_TOKEN");
if (accessToken == null) {
    accessToken = System.getenv("DATABRICKS_TOKEN");
}

String url = "jdbc:databricks://" + host + ":443"
    + ";httpPath=" + httpPath
    + ";AuthMech=11"
    + ";Auth_Flow=0"
    + ";Auth_AccessToken=" + accessToken
    + ";UserAgentEntry=YourCompany_Product/1.0.0";

Connection conn = DriverManager.getConnection(url);

Token Lifetime

Databricks OAuth access tokens typically expire in ~1 hour. The caller is responsible for refreshing.

---

JDBC Connection Pattern (All U2M Flows)

All three U2M flows connect to JDBC the same way: AuthMech=11, Auth_Flow=0, Auth_AccessToken (token pass-through).

[Token Acquisition]           →  [JDBC Connection]
                                 AuthMech=11
SDK or pre-obtained token    →   Auth_Flow=0
                                 Auth_AccessToken=<token>
                                 UserAgentEntry=<partner-ua>

Why SDK + Auth_Flow=0 Instead of JDBC Auth_Flow=2?

The JDBC driver's Auth_Flow=2 (browser-based) has limitations:

• Uses its own built-in OAuth app with a hardcoded redirect port (8020)
• Does not support custom OAuth apps or custom redirect URIs
• For ISV integrations, use the SDK for token acquisition and pass to JDBC via Auth_Flow=0

---

Environment Variables Reference

Flow	Required	Optional
external-browser	DATABRICKS_HOST	DATABRICKS_WAREHOUSE_ID, DATABRICKS_HTTP_PATH. Unset CLIENT_ID/CLIENT_SECRET.
custom-oauth-app	DATABRICKS_HOST, DATABRICKS_U2M_CLIENT_ID	DATABRICKS_REDIRECT_URI (default varies), DATABRICKS_U2M_CLIENT_SECRET, DATABRICKS_WAREHOUSE_ID
token-env	DATABRICKS_HOST, DATABRICKS_ACCESS_TOKEN or DATABRICKS_TOKEN	DATABRICKS_WAREHOUSE_ID, DATABRICKS_HTTP_PATH

---

Java SDK Workarounds (v0.54.0)

When using DatabricksConfig programmatically for external-browser auth:

Workaround	Code	Why	Without It
Set scopes	config.setScopes(Arrays.asList("all-apis"))	SDK calls String.join() on null scopes	NullPointerException at TokenCacheUtils
Call resolve	config.resolve()	Initializes HTTP client	NullPointerException at Consent.<init>
Set redirect URL (built-in)	config.setOAuthRedirectUrl("http://localhost:8020")	Java SDK defaults to :8080/callback	"redirect_uri not registered"

---

Troubleshooting

Error	Cause	Fix
"OAuth application with client_id not available"	Using M2M client_id for browser flow	Use DATABRICKS_U2M_CLIENT_ID or no client_id
"redirect_uri not registered for 'databricks-cli'"	Java SDK default redirect mismatch	Set config.setOAuthRedirectUrl("http://localhost:8020")
"redirect_uri not registered" (custom app)	URI mismatch with registered URI	Ensure exact match in App connections
{"error": "invalid_client"} on token exchange	App is confidential; client_secret missing from POST body	Add client_secret=<secret> to token exchange POST; regenerate if lost
NullPointerException at TokenCacheUtils	Scopes not set	Add config.setScopes(Arrays.asList("all-apis"))
NullPointerException at Consent.<init>	HTTP client not initialized	Add config.resolve()
"more than one authorization method"	Multiple auth env vars	Use env -i to isolate
Token expired (~1 hour)	OAuth access tokens have limited lifetime	Re-run flow or use refresh_token
Script hangs after browser	HTTPServer blocks forever	Set server.timeout
HTTPServer shutdown hangs	Using server.shutdown()	Use server.server_close()
Address already in use	Previous run left port bound	lsof -ti:<port> | xargs kill -9

---

Redirect URI Summary

Scenario	Recommended URI	Register Where
Built-in app (Python SDK)	http://localhost:8020 (default)	Already registered on databricks-cli
Built-in app (Java SDK)	http://localhost:8020 (must set)	Already registered on databricks-cli
Custom OAuth app (local)	http://localhost:8080/callback	App connections → your app
Custom OAuth app (production)	https://your-app.example.com/callback	App connections → your app

---

Go U2M Flows

The Databricks SQL Driver for Go and Databricks SDK for Go support U2M via different mechanisms:

SDK/Driver	U2M Browser	U2M Custom OAuth App	U2M Token-Env
Databricks SQL Driver for Go	u2m.NewAuthenticator() (driver-native, dynamic port)	Manual PKCE → WithAccessToken(token)	WithAccessToken(token)
Databricks SDK for Go	Not built-in (uses cached tokens)	Manual PKCE → Config{Token: token}	Config{Token: token}

Key differences:

• Go SQL Driver's u2m.NewAuthenticator() uses a dynamic localhost port
• Go SDK's native U2M reads cached tokens from databricks auth login
• Both support custom-OAuth-app PKCE via manual implementation

---

Reference

• Authorize user access to Databricks with OAuth (U2M)
• Databricks JDBC OSS driver
• Databricks Java SDK
• Databricks SQL Driver for Go
• Databricks SDK for Go