databricks-isv-adding-databricks-connector

<!-- skill-version: 1.0.0 --> # Adding a Databricks Connector to an Existing Project Use this skill when **your project already exists** and you want to **add Databricks as a new integration** (no Databricks code yet). The other skills cover *how* to implement auth, telemetry, and operations; this one covers *where* to plug in and *what* to build first. **This skill leverages the others:** it does not duplicate auth snippets, validation steps, or driver config. It routes you to the right skill/rule for each concern. When implementing, **read the linked skill or rule** for the chosen stack (e.g. [python-sql-connector](../python-sql-connector/SKILL.md) + [python-sql-connector/authentication.md](../python-sql-connector/authentication.md) for Python SQL). ## Skills and rules leveraged | Concern | Delegates to | |---------|--------------| | Connector shape (config, connect, operations) | [connector-structure](../connector-structure/SKILL.md) | | User input by auth type (PAT / M2M / U2M) | [connector-structure – User input by auth type](../connector-structure/SKILL.md#user-input-by-auth-type) | | U2M flows, M2M vs U2M client_id | [u2m](../u2m/SKILL.md) | | Auth isolation, clean env testing | [testing](../testing/SKILL.md), main [.mdc](../../.cursor/rules/databricks-isv-integration.mdc) | | Validation (UC table + Statement Execution) | [integration-checklist – E2E examples](../connector-testing/integration-checklist.md#using-these-rules-to-build-working-e2e-examples) | | User-Agent format and driver config | [telemetry-attribution](../telemetry-attribution/SKILL.md) | | Stack-specific auth and code | [rest-api](../rest-api/SKILL.md), [python](../python/SKILL.md), [python-sqlalchemy](../python-sqlalchemy/SKILL.md), [databricks-connect](../databricks-connect/SKILL.md), [java-jdbc](../java-jdbc/SKILL.md) + corresponding rules | ## What the rules/skills give you - **Auth:** PAT, OAuth M2M, OAuth U2M (see [connector-structure](../connector-structure/SKILL.md), [u2m](../u2m/SKILL.md), and the rules per stack). - **Telemetry:** User-Agent required; format `<isv>_<product>/<version>`, set in code (not by end user). See [telemetry-attribution](../telemetry-attribution/SKILL.md). - **Connector shape:** Single config, single `connect(config)` entry point, all operations through the returned client. See [connector-structure](../connector-structure/SKILL.md). - **Validation:** UC table get + Statement Execution (or equivalent per stack). See [integration-checklist](../connector-testing/integration-checklist.md). ## Choosing your stack Pick **one** primary path based on your existing project's language and how you need to talk to Databricks: | If your project is… | Prefer | Rule / skill | |---------------------|--------|--------------| | **Any language, HTTP-only** | REST API (token/M2M in headers, Statement Execution + UC APIs) | [rest-api/authentication.md](../rest-api/authentication.md), [rest-api](../rest-api/SKILL.md) | | **Python, need SQL or general API** | Python SDK + optional SQL connector | [python-sdk/authentication.md](../python-sdk/authentication.md), [python-sql-connector/authentication.md](../python-sql-connector/authentication.md), [python-sdk](../python-sdk/SKILL.md), [python-sql-connector](../python-sql-connector/SKILL.md) | | **Python, ORM / SQLAlchemy** | SQLAlchemy + Databricks dialect | [python-sqlalchemy/authentication.md](../python-sqlalchemy/authentication.md), [python-sqlalchemy](../python-sqlalchemy/SKILL.md) | | **Python, Spark / DataFrame API** | Databricks Connect | [databricks-connect/authentication.md](../databricks-connect/authentication.md), [databricks-connect](../databricks-connect/SKILL.md) | | **Java/JVM, JDBC** | OSS JDBC driver (PAT, M2M, U2M) | [java-jdbc/authentication.md](../java-jdbc/authentication.md), [java-jdbc](../java-jdbc/SKILL.md) | Use the chosen rule/skill for exact config fields, auth types, and code snippets. Support **all three auth types** (PAT, M2M, U2M) so users can choose per connection. ## Where to put the integration - **Dedicated module:** Add a package/module for Databricks (e.g. `integrations/databricks/`, `connectors/databricks/`, or `databricks/`) so all Databricks-specific code (config, connect, API/SQL calls) lives in one place. - **Adapter pattern:** If you already have a generic "connection" or "data source" abstraction (e.g. `DataSource`, `ConnectionFactory`), implement a **Databricks adapter** that: - Accepts your existing connection config shape (or a small extension for `host`, `auth_type`, credentials, optional `http_path`/`warehouse_id`). - Implements the same interface as your other connectors (e.g. `connect()`, `execute_query()`, `list_tables()`). - Uses the chosen stack (REST client, Python SDK, JDBC, etc.) internally and sets **User-Agent** and **auth** as described in the connector-structure skill. - **Config schema:** Extend your existing connection/config schema with a "Databricks" type and the parameters from [connector-structure – User input by auth type](../connector-structure/SKILL.md#user-input-by-auth-type). Do **not** expose User-Agent as a user field; set it in code from your product name and version. ## Minimal steps (add Databricks from scratch) 1. **Dependency** – Add the right dependency for your stack (e.g. `databricks-sdk`, `databricks-sql-connector`, `databricks-jdbc`, or use REST with your HTTP client). 2. **Config** – Add a Databricks connection type and fields: `host`, `auth_type` (pat | oauth_m2m | oauth_u2m), and the credentials for that type (token, or client_id + client_secret, or access_token / U2M flow). Optional: `http_path` and/or `warehouse_id` for SQL. See connector-structure for full tables. 3. **connect()** – Implement a single entry point that branches on `auth_type`, builds the client (REST wrapper, `WorkspaceClient`, JDBC `Connection`, etc.), and sets User-Agent (connector-level constant). Use `auth_type="oauth-m2m"` for M2M when using the Python SDK. 4. **Operations** – Route all Databricks operations through this client (no ad-hoc auth or missing User-Agent). If you need SQL, use Statement Execution API (REST) or the SQL connector / JDBC; for metadata, use UC APIs or SDK. 5. **Validation** – Optionally call the two validation tests (UC table get, Statement Execution with a simple query) after connect to verify credentials and permissions. See [integration-checklist – Using these rules to build working E2E examples](../connector-testing/integration-checklist.md#using-these-rules-to-build-working-e2e-examples). 6. **Testing** – Run each auth type (PAT, M2M, U2M) with a **clean env** (only the vars for that auth). See [testing](../testing/SKILL.md). ## Pitfalls to avoid - **M2M vs U2M client_id:** OAuth M2M uses a *service principal* client_id; U2M browser flow uses a *custom OAuth app* client_id (or the SDK built-in app if you use external-browser). Do not reuse the M2M client_id for U2M. See [u2m](../u2m/SKILL.md). - **Auth isolation:** Never set both PAT and M2M env vars in the same process when testing or when the connector chooses auth at runtime. One connection = one auth type. - **User-Agent:** Required on every request; set it in code from your product name/version, not from user input. ## Summary **Yes, the current skills address building a Databricks connector for an existing project:** use this skill to choose stack and placement, then use [connector-structure](../connector-structure/SKILL.md) for config/connect/operations shape, the **authentication.md** file for your chosen stack for auth and telemetry, and [integration-checklist](../connector-testing/integration-checklist.md) for validation and partner requirements.