// Build PWAF-compliant ISV integrations with Databricks: OAuth, telemetry (User-Agent), Unity Catalog, JDBC, SDK, SQL drivers, REST API, Databricks Connect.
| name | databricks-isv-integration |
| description | Build PWAF-compliant ISV integrations with Databricks: OAuth, telemetry (User-Agent), Unity Catalog, JDBC, SDK, SQL drivers, REST API, Databricks Connect. |
This skill provides patterns and code examples for building PWAF-compliant ISV integrations with Databricks, following the Partner Well-Architected Framework (PWAF).
Cursor rule vs skills: Use the Cursor rule (.cursor/rules/databricks-isv-integration.mdc) for file-scoped context when editing integration code (it applies when you work in matching paths). Use the skills in skills/*/SKILL.md for task-based guidance (e.g. "add a Databricks connector," "fix U2M," "run auth tests")—the agent loads the relevant skill by description. Auth patterns live in skills/<connector>/authentication.md, co-located with each skill.
PWAF covers:
All PWAF-validated partner integrations must meet these requirements:
| Requirement | Summary |
|---|---|
| Telemetry | User-Agent tagging on all API/driver calls |
| OAuth | Support Token Federation, U2M with PKCE, and M2M |
| Unity Catalog | Use UC namespaces, respect ACLs, use Volumes for staging |
Applications: Use OAuth M2M for service/backend flows, or U2M for interactive user sign-in. U2M has three flows: (1) SDK external-browser with built-in app (simplest; no custom app), (2) SDK external-browser with custom OAuth app (for ISV integrations; configurable redirect URI), (3) token pass-through (pre-obtained token; headless/CI). All use Auth_Flow=0 on the JDBC side. For custom apps, redirect URI is configurable; default http://localhost:8080/callback. For the built-in app, redirect URI is http://localhost:8020.
import java.sql.*;
import java.util.Properties;
public class DatabricksConnector {
public static Connection connect(
String host,
String httpPath,
String clientId,
String clientSecret,
String userAgent
) throws SQLException {
String url = "jdbc:databricks://" + host + ":443";
Properties props = new Properties();
props.put("httpPath", httpPath);
props.put("SSL", "1");
// OAuth M2M Authentication
props.put("AuthMech", "11");
props.put("Auth_Flow", "1");
props.put("OAuth2ClientId", clientId);
props.put("OAuth2Secret", clientSecret);
// Partner Telemetry (REQUIRED)
props.put("UserAgentEntry", userAgent);
return DriverManager.getConnection(url, props);
}
}
When building a connector UI, show these fields based on auth selection:
| Auth Type | AuthMech | Auth_Flow | Required Fields | Token Source |
|---|---|---|---|---|
| PAT | 3 | - | token | User provides PAT |
| OAuth M2M | 11 | 1 | clientId, clientSecret | Driver does client-credentials |
| U2M Built-in Browser | 11 | 0 | (none beyond host/path) | SDK external-browser (built-in databricks-cli app) |
| U2M Custom App Browser | 11 | 0 | u2mClientId; optional u2mClientSecret, redirectUri | SDK external-browser (custom OAuth app) |
| U2M Token Pass-through | 11 | 0 | accessToken | Pre-obtained (CLI, hosted callback, refresh) |
U2M: All three U2M flows use Auth_Flow=0 (token pass-through) on the JDBC side. The token is obtained externally via the Databricks SDK or a prior OAuth exchange. For custom OAuth apps, use DATABRICKS_U2M_CLIENT_ID (not the M2M DATABRICKS_CLIENT_ID). Redirect URI: configurable for custom apps; http://localhost:8020 for the built-in app (must set explicitly in Java SDK v0.54.0).
See skills/java-jdbc/authentication.md for complete code examples including Java SDK workarounds.
User-Agent Format (required per PWAF for all partner integrations):
<isv-name>_<product-name>/<product-version>
Example: AcmePartner_DataConnector/2.1.0
See skills/telemetry-attribution/SKILL.md for driver-specific configuration.
Quick index: INDEX.md – one line per rule and skill.
skills/<connector>/authentication.md)| File | Language/Driver | Coverage |
|---|---|---|
| skills/java-jdbc/authentication.md | Java JDBC (OSS v3+) | PAT, OAuth M2M, OAuth U2M (browser + token-env), UserAgentEntry, Java 17+ add-opens |
| skills/java-sdk/authentication.md | Java SDK (databricks-sdk-java) | PAT, OAuth M2M; UserAgent.withProduct/withPartner; no warehouse needed |
| skills/python-sdk/authentication.md | Python SDK (databricks-sdk) | PAT, OAuth M2M/U2M, Azure MSI |
| skills/python-sql-connector/authentication.md | Python SQL Connector | PAT, OAuth M2M/U2M |
| skills/python-sqlalchemy/authentication.md | Python SQLAlchemy (databricks-sqlalchemy) | PAT, OAuth M2M/U2M; URL + connect_args user_agent_entry; M2M token from headers dict |
| skills/databricks-connect/authentication.md | Databricks Connect | PAT, OAuth M2M, OAuth U2M (external-browser, localhost, token-env), Serverless, Classic, Azure MSI |
| skills/nodejs-sql-driver/authentication.md | Node.js SQL Driver (@databricks/sql) | PAT, OAuth M2M (driver-native), OAuth U2M (browser), Token pass-through; host normalization; session patterns |
| skills/go-sdk/authentication.md | Databricks SDK for Go (databricks-sdk-go) | PAT, OAuth M2M, OAuth U2M (browser + token-env + custom OAuth app PKCE), Azure MSI; useragent.WithProduct/WithPartner |
| skills/go-sql-driver/authentication.md | Databricks SQL Driver for Go (databricks-sql-go) | PAT, OAuth M2M, OAuth U2M; WithUserAgentEntry; all_auth pattern |
| skills/rest-api/authentication.md | Any language (HTTP) | PAT, OAuth M2M, Token |
| skills/python-dbconnect/authentication.md | Python Databricks Connect | PAT, M2M, U2M; compute resolution; PKCE helper |
| skills/odbc/authentication.md | BI Tools (ODBC) | PAT, OAuth M2M/U2M, Token |
| Path | Purpose |
|---|---|
| .cursor/rules/databricks-isv-integration.mdc | Cursor rule: auth isolation, U2M gotchas, Python SDK/SQL patterns, connector structure, testing |
| skills/adding-databricks-connector/SKILL.md | Add a Databricks connector to an existing project (no Databricks yet): stack choice, where to integrate, minimal steps |
| skills/connector-structure/SKILL.md | How to structure a connector: config, connect(), operations, validation |
| skills/rest-api/SKILL.md | REST API auth (PAT, M2M, U2M) and validation tests |
| skills/python-sdk/SKILL.md | Python SDK (databricks-sdk): Config, auth_type, WorkspaceClient, telemetry |
| skills/python-sql-connector/SKILL.md | Python SQL Connector: PAT, M2M, U2M; credentials_provider; host normalization |
| skills/python-sqlalchemy/SKILL.md | SQLAlchemy + Databricks: dialect, URL, PAT/M2M/U2M, user_agent_entry; M2M token from authenticate() headers |
| skills/databricks-connect/SKILL.md | Databricks Connect: PAT, OAuth M2M, U2M; serverless and classic compute; version compatibility |
| skills/java-jdbc/SKILL.md | Java JDBC (OSS driver): PAT, OAuth M2M, U2M browser/token-env; UserAgentEntry; Java 17+ add-opens |
| skills/java-sdk/SKILL.md | Java SDK (databricks-sdk-java): PAT, OAuth M2M; UserAgent.withProduct/withPartner; UC Tables API; no warehouse |
| skills/go-sdk/SKILL.md | Databricks SDK for Go (databricks-sdk-go): PAT, OAuth M2M, U2M token-env, U2M custom OAuth app (PKCE); useragent.WithProduct/WithPartner; UC Tables API; no warehouse needed |
| skills/go-sql-driver/SKILL.md | Databricks SQL Driver for Go (databricks-sql-go): PAT, OAuth M2M, U2M browser, U2M token-env, U2M custom OAuth app (PKCE); WithUserAgentEntry; DESCRIBE TABLE; SQL warehouse required |
| skills/nodejs-sql-driver/SKILL.md | Node.js SQL Driver (@databricks/sql): PAT, OAuth M2M (driver-native), OAuth U2M (browser); host normalization; session patterns |
| skills/u2m/SKILL.md | U2M flows: external-browser, custom-oauth-app, token-env; M2M client_id ≠ U2M app; separate example scripts per flow |
| skills/testing/SKILL.md | Running auth tests with clean env per test |
Java SDK (workspace APIs, UC, Jobs — no warehouse needed):
<dependency>
<groupId>com.databricks</groupId>
<artifactId>databricks-sdk-java</artifactId>
<version>0.54.0</version>
</dependency>
Java 11+ required. See skills/java-sdk/SKILL.md.
OSS JDBC driver (SQL queries via warehouse):
<dependency>
<groupId>com.databricks</groupId>
<artifactId>databricks-jdbc</artifactId>
<version>3.1.1</version>
<scope>runtime</scope>
</dependency>
On Java 17+, add --add-opens=java.base/java.nio=ALL-UNNAMED (e.g. MAVEN_OPTS or exec plugin). See skills/java-jdbc/SKILL.md.
Databricks SDK for Go (workspace APIs, UC, Jobs — no warehouse needed):
go get github.com/databricks/databricks-sdk-go@v0.107.0
Go 1.21+ required. Uses useragent.WithPartner() / useragent.WithProduct() for telemetry. See skills/go-sdk/SKILL.md.
Databricks SQL Driver for Go (SQL queries via warehouse):
go get github.com/databricks/databricks-sql-go
Go 1.20+ required. Requires DATABRICKS_HTTP_PATH (SQL warehouse). Uses dbsql.WithUserAgentEntry() for telemetry. See skills/go-sql-driver/SKILL.md.
| Issue | Solution |
|---|---|
| More than one authorization method configured | Run with a clean env: only set vars for one auth type (PAT or M2M or U2M). Do not set both DATABRICKS_TOKEN and DATABRICKS_CLIENT_ID/DATABRICKS_CLIENT_SECRET in the same process. |
| OAuth application with client_id not available (U2M) | M2M service principal client_id is not valid for the browser flow. For external-browser, do not pass client_id (SDK uses built-in app). For custom-oauth-app, use a separate OAuth custom app with redirect URI and DATABRICKS_U2M_CLIENT_ID. |
| Cannot configure default credentials (Python M2M) | Pass auth_type="oauth-m2m" to Config(...) when using client_id/client_secret so the SDK does not try default credential resolution. |
| Auth failed with M2M | Verify Service Principal has access to the SQL Warehouse |
| User-Agent not appearing | Check UserAgentEntry is set in connection properties |
| Token expired | OAuth tokens expire in 1 hour; implement refresh logic |
| Unity Catalog access denied | Check catalog/schema grants for the identity |
| Redirect URL for U2M | Custom apps: configurable, default http://localhost:8080/callback. Built-in app: http://localhost:8020 (set explicitly in Java SDK). Register in App connections. Not used for PAT or M2M. |
| Java SDK NullPointerException (U2M) | Call config.setScopes(Arrays.asList("all-apis")) and config.resolve() before config.authenticate(). Set config.setOAuthRedirectUrl("http://localhost:8020") for built-in app. See skills/u2m/SKILL.md. |
| JDBC Java 17+: MemoryUtil / InaccessibleObjectException | OSS driver 3.x uses Arrow; add --add-opens=java.base/java.nio=ALL-UNNAMED to JVM (MAVEN_OPTS or exec plugin). |
PWAF-compliant Python SQL Connector (databricks-sql-connector): PAT, OAuth M2M, OAuth U2M (custom OAuth app PKCE + token-env), credentials_provider patterns, error handling, retry logic. Use when building Python integrations that run SQL queries via a Databricks SQL warehouse.
Add a Databricks connector to an existing project that has no Databricks integration. Use when your product already exists and you want to add Databricks as a new data source or backend.
How to write a build_report.md for any PWAF connector. Captures skill traceability, sufficiency assessment, and test error/fix log. Use after implementing any connector.
How to structure a Databricks connector (REST or Python SDK): config, connect, operations, validation. Use when designing or building a new connector.
How to build and use a PWAF connector test runner (tests/run_all_tests.sh). Covers env isolation, auth types, single connector or single auth mode, parallel execution, browser tests, and report generation.
PWAF-compliant Databricks Connect (Python): PAT, OAuth M2M, OAuth U2M; serverless and classic compute. Use when building or testing Spark-over-Connect integrations.