databricks-isv-adding-databricks-connector

name	databricks-isv-adding-databricks-connector
description	Add a Databricks connector to an existing project that has no Databricks integration. Use when your product already exists and you want to add Databricks as a new data source or backend.

Adding a Databricks Connector to an Existing Project

Use this skill when your project already exists and you want to add Databricks as a new integration (no Databricks code yet). The other skills cover how to implement auth, telemetry, and operations; this one covers where to plug in and what to build first.

This skill leverages the others: it does not duplicate auth snippets, validation steps, or driver config. It routes you to the right skill/rule for each concern. When implementing, read the linked skill or rule for the chosen stack (e.g. python-sql-connector + python-sql-connector/authentication.md for Python SQL).

Skills and rules leveraged

Concern	Delegates to
Connector shape (config, connect, operations)	connector-structure
User input by auth type (PAT / M2M / U2M)	connector-structure – User input by auth type
U2M flows, M2M vs U2M client_id	u2m
Auth isolation, clean env testing	testing, main .mdc
Validation (UC table + Statement Execution)	integration-checklist – E2E examples
User-Agent format and driver config	telemetry-attribution
Stack-specific auth and code	rest-api, python, python-sqlalchemy, databricks-connect, java-jdbc + corresponding rules

What the rules/skills give you

• Auth: PAT, OAuth M2M, OAuth U2M (see connector-structure, u2m, and the rules per stack).
• Telemetry: User-Agent required; format <isv>_<product>/<version>, set in code (not by end user). See telemetry-attribution.
• Connector shape: Single config, single connect(config) entry point, all operations through the returned client. See connector-structure.
• Validation: UC table get + Statement Execution (or equivalent per stack). See integration-checklist.

Choosing your stack

Pick one primary path based on your existing project's language and how you need to talk to Databricks:

If your project is…	Prefer	Rule / skill
Any language, HTTP-only	REST API (token/M2M in headers, Statement Execution + UC APIs)	rest-api/authentication.md, rest-api
Python, need SQL or general API	Python SDK + optional SQL connector	python-sdk/authentication.md, python-sql-connector/authentication.md, python-sdk, python-sql-connector
Python, ORM / SQLAlchemy	SQLAlchemy + Databricks dialect	python-sqlalchemy/authentication.md, python-sqlalchemy
Python, Spark / DataFrame API	Databricks Connect	databricks-connect/authentication.md, databricks-connect
Java/JVM, JDBC	OSS JDBC driver (PAT, M2M, U2M)	java-jdbc/authentication.md, java-jdbc

Use the chosen rule/skill for exact config fields, auth types, and code snippets. Support all three auth types (PAT, M2M, U2M) so users can choose per connection.

Where to put the integration

• Dedicated module: Add a package/module for Databricks (e.g. integrations/databricks/, connectors/databricks/, or databricks/) so all Databricks-specific code (config, connect, API/SQL calls) lives in one place.
• Adapter pattern: If you already have a generic "connection" or "data source" abstraction (e.g. DataSource, ConnectionFactory), implement a Databricks adapter that:
  • Accepts your existing connection config shape (or a small extension for host, auth_type, credentials, optional http_path/warehouse_id).
  • Implements the same interface as your other connectors (e.g. connect(), execute_query(), list_tables()).
  • Uses the chosen stack (REST client, Python SDK, JDBC, etc.) internally and sets User-Agent and auth as described in the connector-structure skill.
• Config schema: Extend your existing connection/config schema with a "Databricks" type and the parameters from connector-structure – User input by auth type. Do not expose User-Agent as a user field; set it in code from your product name and version.

Minimal steps (add Databricks from scratch)

1. Dependency – Add the right dependency for your stack (e.g. databricks-sdk, databricks-sql-connector, databricks-jdbc, or use REST with your HTTP client).
2. Config – Add a Databricks connection type and fields: host, auth_type (pat | oauth_m2m | oauth_u2m), and the credentials for that type (token, or client_id + client_secret, or access_token / U2M flow). Optional: http_path and/or warehouse_id for SQL. See connector-structure for full tables.
3. connect() – Implement a single entry point that branches on auth_type, builds the client (REST wrapper, WorkspaceClient, JDBC Connection, etc.), and sets User-Agent (connector-level constant). Use auth_type="oauth-m2m" for M2M when using the Python SDK.
4. Operations – Route all Databricks operations through this client (no ad-hoc auth or missing User-Agent). If you need SQL, use Statement Execution API (REST) or the SQL connector / JDBC; for metadata, use UC APIs or SDK.
5. Validation – Optionally call the two validation tests (UC table get, Statement Execution with a simple query) after connect to verify credentials and permissions. See integration-checklist – Using these rules to build working E2E examples.
6. Testing – Run each auth type (PAT, M2M, U2M) with a clean env (only the vars for that auth). See testing.

Pitfalls to avoid

• M2M vs U2M client_id: OAuth M2M uses a service principal client_id; U2M browser flow uses a custom OAuth app client_id (or the SDK built-in app if you use external-browser). Do not reuse the M2M client_id for U2M. See u2m.
• Auth isolation: Never set both PAT and M2M env vars in the same process when testing or when the connector chooses auth at runtime. One connection = one auth type.
• User-Agent: Required on every request; set it in code from your product name/version, not from user input.

Summary

Yes, the current skills address building a Databricks connector for an existing project: use this skill to choose stack and placement, then use connector-structure for config/connect/operations shape, the authentication.md file for your chosen stack for auth and telemetry, and integration-checklist for validation and partner requirements.