databricks-isv-integration

# Databricks ISV Integration ## Overview This skill provides patterns and code examples for building **PWAF-compliant ISV integrations** with Databricks, following the [Partner Well-Architected Framework (PWAF)](https://databrickslabs.github.io/partner-architecture/). **Cursor rule vs skills:** Use the **Cursor rule** (`.cursor/rules/databricks-isv-integration.mdc`) for file-scoped context when editing integration code (it applies when you work in matching paths). Use the **skills** in `skills/*/SKILL.md` for task-based guidance (e.g. "add a Databricks connector," "fix U2M," "run auth tests")—the agent loads the relevant skill by description. Auth patterns live in `skills/<connector>/authentication.md`, co-located with each skill. PWAF covers: - **Authentication**: OAuth M2M, U2M, Token Federation, PAT - **Telemetry**: User-Agent attribution for partner tracking - **Unity Catalog**: Proper namespace usage and governance - **SQL Drivers**: JDBC, ODBC, Node.js, Go connectivity ## PWAF Integration Requirements Summary All PWAF-validated partner integrations must meet these requirements: | Requirement | Summary | |-------------|---------| | **Telemetry** | User-Agent tagging on all API/driver calls | | **OAuth** | Support Token Federation, U2M with PKCE, and M2M | | **Unity Catalog** | Use UC namespaces, respect ACLs, use Volumes for staging | **Applications:** Use **OAuth M2M** for service/backend flows, or **U2M** for interactive user sign-in. U2M has three flows: (1) SDK external-browser with built-in app (simplest; no custom app), (2) SDK external-browser with custom OAuth app (for ISV integrations; configurable redirect URI), (3) token pass-through (pre-obtained token; headless/CI). All use `Auth_Flow=0` on the JDBC side. For custom apps, redirect URI is configurable; default `http://localhost:8080/callback`. For the built-in app, redirect URI is `http://localhost:8020`. ## Quick Start: JDBC Connection with OAuth M2M ```java import java.sql.*; import java.util.Properties; public class DatabricksConnector { public static Connection connect( String host, String httpPath, String clientId, String clientSecret, String userAgent ) throws SQLException { String url = "jdbc:databricks://" + host + ":443"; Properties props = new Properties(); props.put("httpPath", httpPath); props.put("SSL", "1"); // OAuth M2M Authentication props.put("AuthMech", "11"); props.put("Auth_Flow", "1"); props.put("OAuth2ClientId", clientId); props.put("OAuth2Secret", clientSecret); // Partner Telemetry (REQUIRED) props.put("UserAgentEntry", userAgent); return DriverManager.getConnection(url, props); } } ``` ## Authentication Types ### Dynamic Field Requirements by Auth Type When building a connector UI, show these fields based on auth selection: | Auth Type | AuthMech | Auth_Flow | Required Fields | Token Source | |-----------|----------|-----------|-----------------|--------------| | PAT | 3 | - | `token` | User provides PAT | | OAuth M2M | 11 | 1 | `clientId`, `clientSecret` | Driver does client-credentials | | U2M Built-in Browser | 11 | 0 | _(none beyond host/path)_ | SDK external-browser (built-in `databricks-cli` app) | | U2M Custom App Browser | 11 | 0 | `u2mClientId`; optional `u2mClientSecret`, `redirectUri` | SDK external-browser (custom OAuth app) | | U2M Token Pass-through | 11 | 0 | `accessToken` | Pre-obtained (CLI, hosted callback, refresh) | **U2M:** All three U2M flows use `Auth_Flow=0` (token pass-through) on the JDBC side. The token is obtained externally via the Databricks SDK or a prior OAuth exchange. For custom OAuth apps, use `DATABRICKS_U2M_CLIENT_ID` (not the M2M `DATABRICKS_CLIENT_ID`). Redirect URI: configurable for custom apps; `http://localhost:8020` for the built-in app (must set explicitly in Java SDK v0.54.0). See [skills/java-jdbc/authentication.md](skills/java-jdbc/authentication.md) for complete code examples including Java SDK workarounds. ## Telemetry Attribution **User-Agent Format** (required per PWAF for all partner integrations): ``` <isv-name>_<product-name>/<product-version> ``` **Example**: `AcmePartner_DataConnector/2.1.0` See [skills/telemetry-attribution/SKILL.md](skills/telemetry-attribution/SKILL.md) for driver-specific configuration. ## Reference Files **Quick index:** [INDEX.md](INDEX.md) – one line per rule and skill. ### Authentication by Driver/SDK (`skills/<connector>/authentication.md`) | File | Language/Driver | Coverage | |------|-----------------|----------| | [skills/java-jdbc/authentication.md](skills/java-jdbc/authentication.md) | Java JDBC (OSS v3+) | PAT, OAuth M2M, OAuth U2M (browser + token-env), UserAgentEntry, Java 17+ add-opens | | [skills/java-sdk/authentication.md](skills/java-sdk/authentication.md) | Java SDK (databricks-sdk-java) | PAT, OAuth M2M; UserAgent.withProduct/withPartner; no warehouse needed | | [skills/python-sdk/authentication.md](skills/python-sdk/authentication.md) | Python SDK (databricks-sdk) | PAT, OAuth M2M/U2M, Azure MSI | | [skills/python-sql-connector/authentication.md](skills/python-sql-connector/authentication.md) | Python SQL Connector | PAT, OAuth M2M/U2M | | [skills/python-sqlalchemy/authentication.md](skills/python-sqlalchemy/authentication.md) | Python SQLAlchemy (databricks-sqlalchemy) | PAT, OAuth M2M/U2M; URL + connect_args user_agent_entry; M2M token from headers dict | | [skills/databricks-connect/authentication.md](skills/databricks-connect/authentication.md) | Databricks Connect | PAT, OAuth M2M, OAuth U2M (external-browser, localhost, token-env), Serverless, Classic, Azure MSI | | [skills/nodejs-sql-driver/authentication.md](skills/nodejs-sql-driver/authentication.md) | Node.js SQL Driver (`@databricks/sql`) | PAT, OAuth M2M (driver-native), OAuth U2M (browser), Token pass-through; host normalization; session patterns | | [skills/go-sdk/authentication.md](skills/go-sdk/authentication.md) | Databricks SDK for Go (`databricks-sdk-go`) | PAT, OAuth M2M, OAuth U2M (browser + token-env + custom OAuth app PKCE), Azure MSI; useragent.WithProduct/WithPartner | | [skills/go-sql-driver/authentication.md](skills/go-sql-driver/authentication.md) | Databricks SQL Driver for Go (`databricks-sql-go`) | PAT, OAuth M2M, OAuth U2M; WithUserAgentEntry; all_auth pattern | | [skills/rest-api/authentication.md](skills/rest-api/authentication.md) | Any language (HTTP) | PAT, OAuth M2M, Token | | [skills/python-dbconnect/authentication.md](skills/python-dbconnect/authentication.md) | Python Databricks Connect | PAT, M2M, U2M; compute resolution; PKCE helper | | [skills/odbc/authentication.md](skills/odbc/authentication.md) | BI Tools (ODBC) | PAT, OAuth M2M/U2M, Token | ### Telemetry & Validation - [skills/telemetry-attribution/SKILL.md](skills/telemetry-attribution/SKILL.md) - User-Agent configuration for all drivers - [skills/connector-testing/integration-checklist.md](skills/connector-testing/integration-checklist.md) - Validation checklist for partners - [skills/connector-testing/env-isolation.md](skills/connector-testing/env-isolation.md) - env -i isolation pattern for tests ### Cursor rule and skills (subfolders) | Path | Purpose | |------|---------| | [.cursor/rules/databricks-isv-integration.mdc](.cursor/rules/databricks-isv-integration.mdc) | Cursor rule: auth isolation, U2M gotchas, Python SDK/SQL patterns, connector structure, testing | | [skills/adding-databricks-connector/SKILL.md](skills/adding-databricks-connector/SKILL.md) | **Add a Databricks connector to an existing project** (no Databricks yet): stack choice, where to integrate, minimal steps | | [skills/connector-structure/SKILL.md](skills/connector-structure/SKILL.md) | How to structure a connector: config, connect(), operations, validation | | [skills/rest-api/SKILL.md](skills/rest-api/SKILL.md) | REST API auth (PAT, M2M, U2M) and validation tests | | [skills/python-sdk/SKILL.md](skills/python-sdk/SKILL.md) | Python SDK (databricks-sdk): Config, auth_type, WorkspaceClient, telemetry | | [skills/python-sql-connector/SKILL.md](skills/python-sql-connector/SKILL.md) | Python SQL Connector: PAT, M2M, U2M; credentials_provider; host normalization | | [skills/python-sqlalchemy/SKILL.md](skills/python-sqlalchemy/SKILL.md) | SQLAlchemy + Databricks: dialect, URL, PAT/M2M/U2M, user_agent_entry; M2M token from authenticate() headers | | [skills/databricks-connect/SKILL.md](skills/databricks-connect/SKILL.md) | Databricks Connect: PAT, OAuth M2M, U2M; serverless and classic compute; version compatibility | | [skills/java-jdbc/SKILL.md](skills/java-jdbc/SKILL.md) | Java JDBC (OSS driver): PAT, OAuth M2M, U2M browser/token-env; UserAgentEntry; Java 17+ add-opens | | [skills/java-sdk/SKILL.md](skills/java-sdk/SKILL.md) | Java SDK (databricks-sdk-java): PAT, OAuth M2M; UserAgent.withProduct/withPartner; UC Tables API; no warehouse | | [skills/go-sdk/SKILL.md](skills/go-sdk/SKILL.md) | Databricks SDK for Go (`databricks-sdk-go`): PAT, OAuth M2M, U2M token-env, U2M custom OAuth app (PKCE); useragent.WithProduct/WithPartner; UC Tables API; no warehouse needed | | [skills/go-sql-driver/SKILL.md](skills/go-sql-driver/SKILL.md) | Databricks SQL Driver for Go (`databricks-sql-go`): PAT, OAuth M2M, U2M browser, U2M token-env, U2M custom OAuth app (PKCE); WithUserAgentEntry; DESCRIBE TABLE; SQL warehouse required | | [skills/nodejs-sql-driver/SKILL.md](skills/nodejs-sql-driver/SKILL.md) | Node.js SQL Driver (`@databricks/sql`): PAT, OAuth M2M (driver-native), OAuth U2M (browser); host normalization; session patterns | | [skills/u2m/SKILL.md](skills/u2m/SKILL.md) | U2M flows: external-browser, custom-oauth-app, token-env; M2M client_id ≠ U2M app; separate example scripts per flow | | [skills/testing/SKILL.md](skills/testing/SKILL.md) | Running auth tests with clean env per test | ## Maven Dependencies (Java) **Java SDK** (workspace APIs, UC, Jobs — no warehouse needed): ```xml <dependency> <groupId>com.databricks</groupId> <artifactId>databricks-sdk-java</artifactId> <version>0.54.0</version> </dependency> ``` Java 11+ required. See [skills/java-sdk/SKILL.md](skills/java-sdk/SKILL.md). **OSS JDBC driver** (SQL queries via warehouse): ```xml <dependency> <groupId>com.databricks</groupId> <artifactId>databricks-jdbc</artifactId> <version>3.1.1</version> <scope>runtime</scope> </dependency> ``` On Java 17+, add `--add-opens=java.base/java.nio=ALL-UNNAMED` (e.g. `MAVEN_OPTS` or exec plugin). See [skills/java-jdbc/SKILL.md](skills/java-jdbc/SKILL.md). ## Go Module Dependencies **Databricks SDK for Go** (workspace APIs, UC, Jobs — no warehouse needed): ``` go get github.com/databricks/databricks-sdk-go@v0.107.0 ``` Go 1.21+ required. Uses `useragent.WithPartner()` / `useragent.WithProduct()` for telemetry. See [skills/go-sdk/SKILL.md](skills/go-sdk/SKILL.md). **Databricks SQL Driver for Go** (SQL queries via warehouse): ``` go get github.com/databricks/databricks-sql-go ``` Go 1.20+ required. Requires `DATABRICKS_HTTP_PATH` (SQL warehouse). Uses `dbsql.WithUserAgentEntry()` for telemetry. See [skills/go-sql-driver/SKILL.md](skills/go-sql-driver/SKILL.md). ## Common Issues | Issue | Solution | |-------|----------| | **More than one authorization method configured** | Run with a clean env: only set vars for one auth type (PAT or M2M or U2M). Do not set both `DATABRICKS_TOKEN` and `DATABRICKS_CLIENT_ID`/`DATABRICKS_CLIENT_SECRET` in the same process. | | **OAuth application with client_id not available (U2M)** | M2M service principal client_id is not valid for the browser flow. For external-browser, do not pass client_id (SDK uses built-in app). For custom-oauth-app, use a separate OAuth custom app with redirect URI and `DATABRICKS_U2M_CLIENT_ID`. | | **Cannot configure default credentials (Python M2M)** | Pass `auth_type="oauth-m2m"` to `Config(...)` when using client_id/client_secret so the SDK does not try default credential resolution. | | **Auth failed with M2M** | Verify Service Principal has access to the SQL Warehouse | | **User-Agent not appearing** | Check `UserAgentEntry` is set in connection properties | | **Token expired** | OAuth tokens expire in 1 hour; implement refresh logic | | **Unity Catalog access denied** | Check catalog/schema grants for the identity |