databricks-isv-databricks-connect

<!-- skill-version: 1.0.0 --> # Databricks Connect (ISV) Use this skill when implementing or testing **Databricks Connect** (Python) for PWAF-compliant partner integrations: remote Spark execution from external apps using `databricks-connect` and `DatabricksSession`. ## Reference - **Full patterns:** [authentication.md](authentication.md) – PAT, OAuth M2M, OAuth U2M (external-browser, localhost, token-env), serverless and classic compute, User-Agent, version compatibility, troubleshooting. ## Golden snippets (copy-paste accurate) **Serverless PAT – Config and session with .userAgent():** ```python from databricks.connect import DatabricksSession from databricks.sdk.config import Config config = Config( host=host_url, token=token, serverless_compute_id="auto", ) session = ( DatabricksSession.builder .sdkConfig(config) .userAgent("YourCompany_YourProduct/1.0.0") .getOrCreate() ) ``` **OAuth M2M – always set auth_type:** ```python config = Config( host=host_url, client_id=client_id, client_secret=client_secret, auth_type="oauth-m2m", # required serverless_compute_id="auto", ) session = ( DatabricksSession.builder .sdkConfig(config) .userAgent("YourCompany_YourProduct/1.0.0") .getOrCreate() ) ``` **U2M – get token first (e.g. external-browser), then Config with token:** Same as PAT but `token=access_token` from browser flow. For external-browser, unset `DATABRICKS_CLIENT_ID`/`DATABRICKS_CLIENT_SECRET` before calling SDK. ## Requirements - **User-Agent (required):** Use `.userAgent("<isv>_<product>/<version>")` on `DatabricksSession.builder` (PWAF-recommended). Do not expose as user config. See [PWAF Databricks Connect telemetry](https://databrickslabs.github.io/partner-architecture/isv-partners/telemetry-attribution/databricks-connect). - **Auth types:** Support PAT, OAuth M2M, and OAuth U2M. One connection = one auth type; do not set both PAT and M2M (or U2M) vars in the same process. - **OAuth M2M:** Always pass `auth_type="oauth-m2m"` to `Config(...)` when using `client_id`/`client_secret` so the SDK does not run default credential resolution. ## Compute options - **Serverless (recommended):** `DATABRICKS_SERVERLESS_COMPUTE_ID=auto` or `Config(..., serverless_compute_id="auto")`. No cluster to manage. Some Connect versions do not yet support serverless; in that case use classic. - **Classic:** `DATABRICKS_CLUSTER_ID` or `CLASSIC_COMPUTE_HTTP_PATH` (cluster ID = last path segment, e.g. `sql/protocolv1/o/<workspace_id>/<cluster_id>`). - **Rule:** Do not set both serverless and classic in the same env; Config errors with "Can't set both cluster id and serverless_compute_id". ## U2M with Connect Obtain an **access token first**, then pass it to `Config(host=..., token=access_token, ...)` for `DatabricksSession`. Same three flows as REST/SQL U2M: | Flow | Use when | Required | |------|----------|----------| | **external-browser** | No custom OAuth app; SDK opens browser | `DATABRICKS_HOST`; unset `DATABRICKS_CLIENT_ID`/`DATABRICKS_CLIENT_SECRET`/`DATABRICKS_TOKEN` so SDK uses built-in app | | **localhost** | Custom OAuth app with localhost redirect | `DATABRICKS_HOST`, `DATABRICKS_CLIENT_ID` (OAuth app, not M2M); optional redirect_uri, client_secret | | **token-env** | Pre-obtained token (no browser) | `DATABRICKS_HOST`, `DATABRICKS_ACCESS_TOKEN` or `DATABRICKS_TOKEN` | **Important:** M2M service principal `client_id` is not valid for browser U2M. For external-browser, do not pass client_id. For localhost, use a separate OAuth custom app. ## Version compatibility - `databricks-connect` version must match the Databricks Runtime (DBR) on compute (e.g. DBR 18 → `databricks-connect==18.0.*`). - Do not install PySpark separately; databricks-connect supplies the matching Spark stack. ## Auth isolation Run each test or script with a **clean environment**: only the vars for the chosen auth type and compute. Use `env -i` plus explicit vars when running tests so PAT, M2M, and U2M do not mix (avoids "more than one authorization method configured"). ## Validation - Build `DatabricksSession` with the chosen auth and compute; run a simple read (e.g. `spark.table("samples.nyctaxi.trips").count()`) to verify connection and permissions.