| name | databricks-isv-adding-databricks-connector |
| description | Add a Databricks connector to an existing project that has no Databricks integration. Use when your product already exists and you want to add Databricks as a new data source or backend. |
Adding a Databricks Connector to an Existing Project
Use this skill when your project already exists and you want to add Databricks as a new integration (no Databricks code yet). The other skills cover how to implement auth, telemetry, and operations; this one covers where to plug in and what to build first.
This skill leverages the others: it does not duplicate auth snippets, validation steps, or driver config. It routes you to the right skill/rule for each concern. When implementing, read the linked skill or rule for the chosen stack (e.g. python-sql-connector + python-sql-connector/authentication.md for Python SQL).
Skills and rules leveraged
What the rules/skills give you
- Auth: PAT, OAuth M2M, OAuth U2M (see connector-structure, u2m, and the rules per stack).
- Telemetry: User-Agent required; format
<isv>_<product>/<version>, set in code (not by end user). See telemetry-attribution.
- Connector shape: Single config, single
connect(config) entry point, all operations through the returned client. See connector-structure.
- Validation: UC table get + Statement Execution (or equivalent per stack). See integration-checklist.
Choosing your stack
Pick one primary path based on your existing project's language and how you need to talk to Databricks:
| If your project is… | Prefer | Rule / skill |
|---|
| Any language, HTTP-only | REST API (token/M2M in headers, Statement Execution + UC APIs) | rest-api/authentication.md, rest-api |
| Python, need SQL or general API | Python SDK + optional SQL connector | python-sdk/authentication.md, python-sql-connector/authentication.md, python-sdk, python-sql-connector |
| Python, ORM / SQLAlchemy | SQLAlchemy + Databricks dialect | python-sqlalchemy/authentication.md, python-sqlalchemy |
| Python, Spark / DataFrame API | Databricks Connect | databricks-connect/authentication.md, databricks-connect |
| Java/JVM, JDBC | OSS JDBC driver (PAT, M2M, U2M) | java-jdbc/authentication.md, java-jdbc |
Use the chosen rule/skill for exact config fields, auth types, and code snippets. Support all three auth types (PAT, M2M, U2M) so users can choose per connection.
Where to put the integration
- Dedicated module: Add a package/module for Databricks (e.g.
integrations/databricks/, connectors/databricks/, or databricks/) so all Databricks-specific code (config, connect, API/SQL calls) lives in one place.
- Adapter pattern: If you already have a generic "connection" or "data source" abstraction (e.g.
DataSource, ConnectionFactory), implement a Databricks adapter that:
- Accepts your existing connection config shape (or a small extension for
host, auth_type, credentials, optional http_path/warehouse_id).
- Implements the same interface as your other connectors (e.g.
connect(), execute_query(), list_tables()).
- Uses the chosen stack (REST client, Python SDK, JDBC, etc.) internally and sets User-Agent and auth as described in the connector-structure skill.
- Config schema: Extend your existing connection/config schema with a "Databricks" type and the parameters from connector-structure – User input by auth type. Do not expose User-Agent as a user field; set it in code from your product name and version.
Minimal steps (add Databricks from scratch)
- Dependency – Add the right dependency for your stack (e.g.
databricks-sdk, databricks-sql-connector, databricks-jdbc, or use REST with your HTTP client).
- Config – Add a Databricks connection type and fields:
host, auth_type (pat | oauth_m2m | oauth_u2m), and the credentials for that type (token, or client_id + client_secret, or access_token / U2M flow). Optional: http_path and/or warehouse_id for SQL. See connector-structure for full tables.
- connect() – Implement a single entry point that branches on
auth_type, builds the client (REST wrapper, WorkspaceClient, JDBC Connection, etc.), and sets User-Agent (connector-level constant). Use auth_type="oauth-m2m" for M2M when using the Python SDK.
- Operations – Route all Databricks operations through this client (no ad-hoc auth or missing User-Agent). If you need SQL, use Statement Execution API (REST) or the SQL connector / JDBC; for metadata, use UC APIs or SDK.
- Validation – Optionally call the two validation tests (UC table get, Statement Execution with a simple query) after connect to verify credentials and permissions. See integration-checklist – Using these rules to build working E2E examples.
- Testing – Run each auth type (PAT, M2M, U2M) with a clean env (only the vars for that auth). See testing.
Pitfalls to avoid
- M2M vs U2M client_id: OAuth M2M uses a service principal client_id; U2M browser flow uses a custom OAuth app client_id (or the SDK built-in app if you use external-browser). Do not reuse the M2M client_id for U2M. See u2m.
- Auth isolation: Never set both PAT and M2M env vars in the same process when testing or when the connector chooses auth at runtime. One connection = one auth type.
- User-Agent: Required on every request; set it in code from your product name/version, not from user input.
Summary
Yes, the current skills address building a Databricks connector for an existing project: use this skill to choose stack and placement, then use connector-structure for config/connect/operations shape, the authentication.md file for your chosen stack for auth and telemetry, and integration-checklist for validation and partner requirements.
