databricks-isv-connector-structure

name	databricks-isv-connector-structure
description	How to structure a Databricks connector (REST or Python SDK): config, connect, operations, validation. Use when designing or building a new connector.

Connector Structure (ISV)

Use this skill when designing or building a Databricks connector (REST-based or Python SDK-based). It defines a consistent shape so auth, telemetry, and operations stay correct.

Recommended auth support

Support all three auth types: PAT, OAuth M2M, and OAuth U2M.

Auth type	Use case	Recommended
PAT	Testing, development, simple automation	Yes – optional for production but required for many workflows.
OAuth M2M	Service-to-service, production backends, automated jobs	Yes – recommended for production.
OAuth U2M	Interactive user sign-in, desktop/UI apps, user context	Yes – app-implemented browser flow or token pass-through.

Design your config and connect() so the user can choose one of these per connection. Do not mix them in a single connection (one connection = one auth type).

User-Agent (required, connector-level)

User-Agent is required and is coded at the connector level. The end user does not provide or configure User-Agent. The connector developer sets it in code (e.g. a constant or build-time value). Format: <isv-name>_<product-name>/<product-version> (e.g. AcmePartner_DataConnector/2.1.0). The connector sends this header on every API/driver call so usage is attributed correctly in Databricks (audit, query history). Do not expose User-Agent (or product/product_version) as a user-configurable connection parameter.

Reference

• Adding a Databricks connector to an existing project (no Databricks yet): skills/adding-databricks-connector/SKILL.md – stack choice, where to integrate, minimal steps.
• Auth patterns: skills/rest-api/authentication.md, skills/python-sdk/authentication.md, skills/python-sql-connector/authentication.md, skills/python-sqlalchemy/authentication.md (SQLAlchemy dialect), skills/databricks-connect/authentication.md (Databricks Connect), skills/java-jdbc/authentication.md (Java JDBC OSS: PAT, M2M, U2M; UserAgentEntry).
• Environment isolation: skills/connector-testing/env-isolation.md — read before testing. Covers the DATABRICKS_AUTH_TYPE trap (breaks Python SDK, Go SDK, Java SDK), ~/.databrickscfg DEFAULT profile leakage, and the env -i isolation pattern for tests.
• Checklist: skills/connector-testing/integration-checklist.md.

---

1. Config / connection parameters

Use a single config object (or connection params) that includes:

Parameter	Required	Description
host	Yes	Workspace URL (e.g. https://myworkspace.cloud.databricks.com). End user provides.
auth_type	Yes	One of: pat | oauth_m2m | oauth_u2m | token (pass-through). Recommend supporting all three: PAT, M2M, U2M.
Credentials by auth_type	Yes	For PAT: token. For M2M: client_id, client_secret. For U2M: access_token (or run browser flow once and pass token). End user provides (except when connector runs browser flow).
warehouse_id	If using SQL	Required for Statement Execution API. End user provides – or the connector can extract it from http_path (e.g. path /sql/1.0/warehouses/abc123 → warehouse_id abc123).
http_path	If using SQL connector	SQL Warehouse HTTP path (e.g. /sql/1.0/warehouses/abc123). End user provides. The connector can derive warehouse_id from this path if needed for the Statement Execution API.
redirect_uri	Optional (U2M localhost)	Default e.g. http://localhost:8080/callback. End user may override.
Databricks Connect only
serverless_compute_id	For Connect serverless	Set to "auto" (recommended when supported). End user may enable.
cluster_id or classic_compute_http_path	For Connect classic	Cluster ID or HTTP path (e.g. sql/protocolv1/o/<workspace_id>/<cluster_id>). End user provides. One of serverless or classic per connection.

User-Agent: Not a user parameter. The connector sets it in code (connector-level); see "User-Agent (required, connector-level)" above.

Rule: One connection = one auth type. Do not accept or mix multiple auth methods in a single config (e.g. do not set both PAT and M2M on the same connection).

---

User input by auth type

All options the end user must (or may) enter for each auth type. User-Agent is not a user input – it is coded at the connector level (see "User-Agent (required, connector-level)" above). If the connector runs SQL or Statement Execution, the user may need to provide warehouse_id and/or http_path. Note: The connector can extract warehouse_id from http_path (e.g. /sql/1.0/warehouses/abc123 → abc123), so the user need not provide both if they already supply http_path.

PAT

Option	Required	Description
host	Yes	Databricks workspace URL (e.g. https://myworkspace.cloud.databricks.com).
token	Yes	Personal access token (starts with dapi...). User creates in workspace: Settings → Developer → Access tokens.
warehouse_id	If using SQL / Statement Execution	SQL Warehouse ID. Can be extracted from http_path if user provides that instead (e.g. /sql/1.0/warehouses/abc123 → abc123).
http_path	If using SQL connector	SQL Warehouse HTTP path (e.g. /sql/1.0/warehouses/abc123). Connector can derive warehouse_id from this.

OAuth M2M (client credentials)

Option	Required	Description
host	Yes	Databricks workspace URL.
client_id	Yes	Service principal application (client) ID (UUID). Created in workspace: Settings → Identity and access → Service principals.
client_secret	Yes	OAuth client secret for that service principal (generate in Service principals UI).
warehouse_id	If using SQL / Statement Execution	SQL Warehouse ID. Can be extracted from http_path if provided.
http_path	If using SQL connector	SQL Warehouse HTTP path. Connector can derive warehouse_id from this.

OAuth U2M (user sign-in)

U2M can be implemented in three ways. Collect only the options for the flow you support.

Option A – External browser (no custom OAuth app)
User signs in via browser; connector uses Databricks built-in OAuth app.

Option	Required	Description
host	Yes	Databricks workspace URL.
warehouse_id	If using SQL / Statement Execution	SQL Warehouse ID. Can be extracted from http_path if provided.
http_path	If using SQL connector	SQL Warehouse HTTP path. Connector can derive warehouse_id from this.

Do not ask for or use client_id / client_secret for this flow (M2M service principal client_id is not valid for browser OAuth).

Option B – Localhost (custom OAuth app)
Admin creates an OAuth app in the account with a localhost redirect URI; user signs in and is redirected back to the connector.

Option	Required	Description
host	Yes	Databricks workspace URL.
client_id	Yes	OAuth application client ID from the custom app (Settings → Developer → App connections). Not the M2M service principal client_id.
redirect_uri	Optional	Redirect URI registered for the app. Default: http://localhost:8080/callback.
client_secret	If app is confidential	OAuth app secret, if the app has one.
warehouse_id	If using SQL / Statement Execution	SQL Warehouse ID. Can be extracted from http_path if provided.
http_path	If using SQL connector	SQL Warehouse HTTP path. Connector can derive warehouse_id from this.

Option C – Token pass-through (pre-obtained token)
User provides an access token already obtained (e.g. from a hosted callback or refresh).

Option	Required	Description
host	Yes	Databricks workspace URL.
access_token	Yes	OAuth access token (from browser flow or refresh).
warehouse_id	If using SQL / Statement Execution	SQL Warehouse ID. Can be extracted from http_path if provided.
http_path	If using SQL connector	SQL Warehouse HTTP path. Connector can derive warehouse_id from this.

---

2. Single entry point: connect(config)

Expose one way to create a connected client:

• REST connector: connect(config) -> Client, where the client holds the resolved token (or M2M token fetcher), host, and User-Agent. All subsequent API calls use this client so headers are consistent.
• Python SDK connector: connect(config) -> WorkspaceClient (or DatabricksSession for Connect). Build Config once from config with the correct auth_type (e.g. auth_type="oauth-m2m" for M2M). Set User-Agent (product/product_version) from connector-level constants, not from user config; return WorkspaceClient(config=...).

Inside connect():

1. Branch on auth_type.
2. For PAT: use token as Bearer (REST) or WorkspaceClient(host=..., token=..., product=..., product_version=...) (SDK). Use connector-level product/product_version for User-Agent, not from user config.
3. For M2M: obtain token (REST: call /oidc/v1/token, cache with expiry; SDK: Config(host=..., client_id=..., client_secret=..., auth_type="oauth-m2m", product=..., product_version=...)). Use connector-level product/product_version.
4. For U2M/token: use provided access_token as Bearer (REST) or WorkspaceClient(host=..., token=access_token, ...) (SDK). If the connector runs the browser flow, do that once and then store the access_token (and optionally refresh_token) in config or session. User-Agent from connector level.

Rule: Do not mix auth methods in one process. If the app supports multiple connection types, each connection instance should use only one auth type (see auth isolation in the main cursor rule).

---

3. Operations through the client

All API or SDK operations go through the client returned by connect():

• REST: Every request uses the client’s Authorization: Bearer <token>, User-Agent, and Content-Type where needed. For M2M, the client refreshes the token when expired (same pattern as in rest-api-authentication.md).
• Python SDK: Use client.tables.get(), client.jobs.list(), etc. No extra auth or User-Agent wiring per call.
• Databricks Connect: Use DatabricksSession.builder.sdkConfig(config).getOrCreate() with a single Config built from host, auth (token or client_id/client_secret with auth_type="oauth-m2m" or U2M token), and compute (serverless_compute_id="auto" or cluster_id). Set product/product_version on Config. Do not set both serverless and classic in the same Config.

This keeps telemetry and auth in one place and avoids mistakes (e.g. missing User-Agent or wrong token).

---

4. Optional: validate_connection()

To verify credentials after connect, run the two recommended validation tests:

1. UC table by full name: GET /api/2.1/unity-catalog/tables/<catalog>.<schema>.<table> (no warehouse). Or with SDK: client.tables.get(full_name=...).
2. Statement Execution: POST /api/2.0/sql/statements with warehouse_id and a simple SQL (e.g. SELECT 1 or DESCRIBE EXTENDED catalog.schema.table AS JSON). Requires warehouse_id in config.

If either fails with 401/403, the connection is invalid or missing permissions. Optionally expose this as client.validate_connection() or a standalone helper.

---

5. Token refresh and secrets

• M2M: Cache the access_token and expiry; before expiry (e.g. 5 minutes), call the token endpoint again and update the cache. See REST rule and integration checklist.
• U2M: When the access_token expires, use the refresh_token with the OAuth token endpoint (grant_type=refresh_token) to get a new access_token. Do not send the refresh_token on every API request.
• Secrets: Never log tokens or secrets. Prefer env vars or a secret store for credentials; see integration checklist (“Tokens never logged”).

---

6. Summary

Layer	Responsibility
User-Agent	Required, connector-level. Set on every request; format <isv>_<product>/<version>. Coded in the connector (not a user-configurable option).
Config	host, auth_type (recommend PAT + M2M + U2M), credentials for that type (see “User input by auth type” above), optional warehouse_id/http_path. No user-supplied User-Agent.
connect(config)	Single entry point; branch on auth_type; return REST client or WorkspaceClient with correct auth and User-Agent.
Client	All operations; consistent headers (REST) or SDK client; M2M token refresh inside client.
validate_connection()	Optional; run UC table get + Statement Execution to verify.
Token refresh	M2M: refresh before expiry; U2M: refresh_token → new access_token. Never log tokens.

Recommendation: Support all three auth types (PAT, OAuth M2M, OAuth U2M) so users can choose the right one per connection. User-Agent is required but is set at the connector level (in code), not by the end user; the end user provides only host, credentials for the chosen auth type, and optional warehouse_id/http_path. Following this structure keeps auth isolation, telemetry, and validation aligned with the REST and Python auth rules and the integration checklist.