databricks-isv-java-jdbc

name	databricks-isv-java-jdbc
description	PWAF-compliant Databricks JDBC Driver (databricks-jdbc v3+): PAT, OAuth M2M, OAuth U2M (custom OAuth app, token-env). Use when building or testing Java SQL warehouse integrations.

Databricks JDBC Driver (ISV)

Use this skill when implementing or testing Databricks JDBC Driver (databricks-jdbc v3+) connectivity for PWAF-compliant SQL warehouse integrations.

PWAF Documentation Links

Resource	URL
PWAF Home	https://databrickslabs.github.io/partner-architecture/
Authentication Best Practices	https://databrickslabs.github.io/partner-architecture/isv-partners/lakehouse-patterns/access-auth/
OAuth M2M Guide	https://databrickslabs.github.io/partner-architecture/isv-partners/lakehouse-patterns/access-auth/oauth-m2m
OAuth U2M Guide	https://databrickslabs.github.io/partner-architecture/isv-partners/lakehouse-patterns/access-auth/oauth-u2m
User-Agent Telemetry	https://databrickslabs.github.io/partner-architecture/isv-partners/telemetry-attribution/sdks
JDBC OSS Docs	https://docs.databricks.com/en/integrations/jdbc-oss/
JDBC GitHub	https://github.com/databricks/databricks-jdbc

Requirements

• Driver: com.databricks:databricks-jdbc:3.1.1 (or later 3.x) from Maven Central. OSS driver only.
• SDK: com.databricks:databricks-sdk-java:0.54.0 – Required for U2M browser-based token acquisition. Not needed for PAT/M2M/token-env.
• Java: 11+ required. Java 17+ needs --add-opens=java.base/java.nio=ALL-UNNAMED.
• User-Agent (required): Set UserAgentEntry on every connection URL (format <isv>_<product>/<version>).

Authentication Decision Guide

Which authentication method to use?

Production / automated workloads?
  → OAuth M2M (Auth_Flow=1)  ✅ RECOMMENDED

User-interactive (ISV custom OAuth app)?
  → U2M Custom OAuth App (SDK → Auth_Flow=0)  ✅ SUPPORTED

Already have an OAuth access token?
  → U2M Token-Env (Auth_Flow=0)  ✅ SUPPORTED

Local development/testing only?
  → PAT (AuthMech=3)  ⚠️ LIMITED

Note: These examples do NOT use the SDK's built-in databricks-cli OAuth app. ISV/partner applications should register custom OAuth apps in App connections for proper branding, audit trails, and scoped permissions.

Authentication Comparison

Method	PWAF Status	JDBC Auth	Token Source	Browser
PAT	⚠️ Limited	AuthMech=3	User provides	No
OAuth M2M	✅ Recommended	Auth_Flow=1	Driver handles	No
U2M Custom OAuth App	✅ Supported	Auth_Flow=0	Java SDK (custom app)	Yes
U2M Token-Env	✅ Supported	Auth_Flow=0	Pre-obtained from env	No

Auth Mapping (JDBC Parameters)

Auth	JDBC Parameters	Token Source
PAT	AuthMech=3, UID=token, PWD=<token>	User provides PAT directly
OAuth M2M	AuthMech=11, Auth_Flow=1, OAuth2ClientId, OAuth2Secret	JDBC driver does client-credentials
OAuth U2M (custom app)	AuthMech=11, Auth_Flow=0, Auth_AccessToken	Java SDK external-browser (custom OAuth app)
OAuth U2M (token-env)	AuthMech=11, Auth_Flow=0, Auth_AccessToken	Pre-obtained from env variable

Both U2M flows use the same JDBC parameters (Auth_Flow=0 token pass-through). The difference is how the token is obtained.

CLIENT_ID Distinction (CRITICAL)

Variable	Purpose	Used By
DATABRICKS_CLIENT_ID	M2M service principal UUID	OAuth M2M only
DATABRICKS_U2M_CLIENT_ID	Custom OAuth app client ID	U2M custom OAuth app

Using the wrong client_id causes: "OAuth application with client_id not available in Databricks account"

Why SDK + Auth_Flow=0 Instead of JDBC Auth_Flow=2?

The JDBC driver's native Auth_Flow=2 uses a built-in OAuth app with a hardcoded redirect port (8020). It does not support custom OAuth apps or custom redirect URIs.

For ISV integrations that need custom OAuth apps, the proven pattern is:

1. Use the Databricks Java SDK (DatabricksConfig with authType="external-browser") to obtain the token
2. Pass the token to the JDBC driver via Auth_Flow=0 (Auth_AccessToken)

This approach supports custom OAuth apps with configurable redirect URIs, which is required for proper ISV branding and audit trails.

Compute / Target

• SQL warehouse only: Connection uses httpPath (e.g., /sql/1.0/warehouses/<id>). The JDBC driver does not support jobs compute.
• Config: Require either DATABRICKS_HTTP_PATH or DATABRICKS_WAREHOUSE_ID (path derived as /sql/1.0/warehouses/<id>).

---

User-Agent (Required per PWAF)

Set on every JDBC connection URL:

String url = "jdbc:databricks://" + host + ":443"
    + ";httpPath=" + httpPath
    + ";AuthMech=3;UID=token;PWD=" + token
    + ";UserAgentEntry=YourCompany_YourProduct/1.0.0";

For SDK operations, also register static telemetry:

import com.databricks.sdk.core.UserAgent;

UserAgent.withProduct("YourCompany_YourProduct", "1.0.0");
UserAgent.withPartner("YourCompany");

---

Environment Variables Reference

Variable	Required For	Description
DATABRICKS_HOST	All	Workspace URL (e.g., https://myworkspace.cloud.databricks.com)
DATABRICKS_HTTP_PATH	All	SQL warehouse HTTP path (e.g., /sql/1.0/warehouses/abc123)
DATABRICKS_WAREHOUSE_ID	Alternative	Warehouse ID (path derived as /sql/1.0/warehouses/<id>)
DATABRICKS_TOKEN	PAT	Personal access token
DATABRICKS_CLIENT_ID	OAuth M2M	Service principal UUID
DATABRICKS_CLIENT_SECRET	OAuth M2M	Service principal OAuth secret
DATABRICKS_U2M_CLIENT_ID	U2M Custom OAuth	Custom OAuth app client ID from App connections
DATABRICKS_U2M_CLIENT_SECRET	U2M Custom OAuth	Custom OAuth app client secret (Java SDK requires this)
DATABRICKS_REDIRECT_URI	U2M Custom OAuth (optional)	Custom redirect URI (default: http://localhost:8080/callback)
DATABRICKS_ACCESS_TOKEN	U2M Token-Env	Pre-obtained OAuth access token
DATABRICKS_AUTH_TYPE	Multi-auth	Auth type selector: pat, oauth_m2m, u2m_custom_oauth_app, u2m_token_env

Important: Do not mix M2M and U2M environment variables. Use env -i for clean test environments.

---

Complete Examples

PAT Authentication (Testing Only)

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;

public class JdbcPatExample {
    public static void main(String[] args) throws Exception {
        String host = System.getenv("DATABRICKS_HOST");
        String token = System.getenv("DATABRICKS_TOKEN");
        String httpPath = System.getenv("DATABRICKS_HTTP_PATH");

        // Normalize host
        host = host.replaceFirst("^https://", "").replaceFirst("^http://", "");

        String url = "jdbc:databricks://" + host + ":443"
            + ";httpPath=" + httpPath
            + ";AuthMech=3"
            + ";UID=token"
            + ";PWD=" + token
            + ";UserAgentEntry=YourCompany_YourProduct/1.0.0";

        try (Connection conn = DriverManager.getConnection(url);
             Statement stmt = conn.createStatement();
             ResultSet rs = stmt.executeQuery("SELECT 1 as test")) {
            if (rs.next()) {
                System.out.println("PAT OK: " + rs.getInt("test"));
            }
        }
    }
}

Env vars: DATABRICKS_HOST, DATABRICKS_TOKEN, DATABRICKS_HTTP_PATH

---

OAuth M2M Authentication (Production Recommended)

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;

public class JdbcOAuthM2MExample {
    public static void main(String[] args) throws Exception {
        String host = System.getenv("DATABRICKS_HOST");
        String clientId = System.getenv("DATABRICKS_CLIENT_ID");
        String clientSecret = System.getenv("DATABRICKS_CLIENT_SECRET");
        String httpPath = System.getenv("DATABRICKS_HTTP_PATH");

        host = host.replaceFirst("^https://", "").replaceFirst("^http://", "");

        String url = "jdbc:databricks://" + host + ":443"
            + ";httpPath=" + httpPath
            + ";AuthMech=11"
            + ";Auth_Flow=1"
            + ";OAuth2ClientId=" + clientId
            + ";OAuth2Secret=" + clientSecret
            + ";UserAgentEntry=YourCompany_YourProduct/1.0.0";

        try (Connection conn = DriverManager.getConnection(url);
             Statement stmt = conn.createStatement();
             ResultSet rs = stmt.executeQuery("SELECT 1 as test")) {
            if (rs.next()) {
                System.out.println("OAuth M2M OK: " + rs.getInt("test"));
            }
        }
    }
}

Env vars: DATABRICKS_HOST, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET, DATABRICKS_HTTP_PATH

Setup: Create service principal in Account Console → Settings → Service principals. Generate OAuth secret.

---

U2M Custom OAuth App (SDK → JDBC)

Uses a custom OAuth app registered in App connections:

import com.databricks.sdk.core.DatabricksConfig;
import com.databricks.sdk.core.UserAgent;
import java.sql.Connection;
import java.sql.DriverManager;
import java.util.Arrays;
import java.util.Map;

public class JdbcU2MCustomOAuthAppExample {
    public static void main(String[] args) throws Exception {
        String host = System.getenv("DATABRICKS_HOST");
        String clientId = System.getenv("DATABRICKS_U2M_CLIENT_ID");
        String clientSecret = System.getenv("DATABRICKS_U2M_CLIENT_SECRET"); // required by Java SDK
        String redirectUri = System.getenv("DATABRICKS_REDIRECT_URI");
        String httpPath = System.getenv("DATABRICKS_HTTP_PATH");

        UserAgent.withProduct("YourCompany_YourProduct", "1.0.0");
        UserAgent.withPartner("YourCompany");

        // Step 1: Get token via SDK with custom OAuth app
        String normalizedHost = host.startsWith("https://") ? host : "https://" + host;
        DatabricksConfig config = new DatabricksConfig()
                .setHost(normalizedHost)
                .setAuthType("external-browser")
                .setClientId(clientId)
                .setScopes(Arrays.asList("all-apis"));

        if (clientSecret != null && !clientSecret.isBlank()) {
            config.setClientSecret(clientSecret);
        }
        if (redirectUri != null && !redirectUri.isBlank()) {
            config.setOAuthRedirectUrl(redirectUri);
        }
        config.resolve();
        Map<String, String> headers = config.authenticate();
        String token = headers.get("Authorization").substring("Bearer ".length());

        // Step 2: Connect via JDBC
        String hostNorm = host.replaceFirst("^https://", "").replaceFirst("^http://", "");
        String url = "jdbc:databricks://" + hostNorm + ":443"
            + ";httpPath=" + httpPath
            + ";AuthMech=11;Auth_Flow=0;Auth_AccessToken=" + token
            + ";UserAgentEntry=YourCompany_YourProduct/1.0.0";

        try (Connection conn = DriverManager.getConnection(url)) {
            System.out.println("U2M Custom OAuth App OK");
        }
    }
}

Env vars: DATABRICKS_HOST, DATABRICKS_U2M_CLIENT_ID, DATABRICKS_HTTP_PATH

Recommended: DATABRICKS_U2M_CLIENT_SECRET (required by Java SDK for proper auth configuration)

Optional: DATABRICKS_REDIRECT_URI (must match App connections registration)

CRITICAL: Use DATABRICKS_U2M_CLIENT_ID for custom OAuth apps — NOT DATABRICKS_CLIENT_ID (that's for M2M).

Note: Testing showed the Java SDK requires DATABRICKS_U2M_CLIENT_SECRET for custom OAuth apps to properly configure external-browser auth. Without it, you may get "cannot configure default credentials" errors.

---

U2M Token-Env (Pre-obtained Token)

For headless/CI environments with a pre-obtained token:

import java.sql.Connection;
import java.sql.DriverManager;

public class JdbcU2MTokenEnvExample {
    public static void main(String[] args) throws Exception {
        String host = System.getenv("DATABRICKS_HOST");
        String httpPath = System.getenv("DATABRICKS_HTTP_PATH");
        String accessToken = System.getenv("DATABRICKS_ACCESS_TOKEN");
        if (accessToken == null || accessToken.isBlank()) {
            accessToken = System.getenv("DATABRICKS_TOKEN");
        }

        String hostNorm = host.replaceFirst("^https://", "").replaceFirst("^http://", "");
        String url = "jdbc:databricks://" + hostNorm + ":443"
            + ";httpPath=" + httpPath
            + ";AuthMech=11"
            + ";Auth_Flow=0"
            + ";Auth_AccessToken=" + accessToken
            + ";UserAgentEntry=YourCompany_YourProduct/1.0.0";

        try (Connection conn = DriverManager.getConnection(url)) {
            System.out.println("U2M Token-Env OK");
        }
    }
}

Env vars: DATABRICKS_HOST, DATABRICKS_HTTP_PATH, DATABRICKS_ACCESS_TOKEN (or DATABRICKS_TOKEN)

---

Token Refresh for U2M Flows

U2M access tokens expire after approximately 1 hour. For long-running JDBC applications, you need to handle token refresh and reconnection.

Token Lifetime Summary

Auth Type	Token TTL	Refresh Strategy
PAT	User-configured (90 days default)	Generate new token manually
OAuth M2M	~1 hour	JDBC driver handles automatically
U2M Custom OAuth App	~1 hour	Re-authenticate via SDK, reconnect JDBC
U2M Token-Env	~1 hour	Application must handle

JDBC Token Refresh Pattern

Since JDBC connections use a token at connection time, you need to obtain a fresh token and create a new connection when tokens expire:

public class JdbcTokenRefreshExample {
    private DatabricksConfig sdkConfig;
    private Connection jdbcConnection;
    private long tokenExpiresAt;
    private static final long REFRESH_BUFFER_MS = 5 * 60 * 1000; // 5 minutes

    public JdbcTokenRefreshExample() throws Exception {
        this.sdkConfig = createSdkConfig();
        this.jdbcConnection = createConnection();
        this.tokenExpiresAt = System.currentTimeMillis() + (55 * 60 * 1000);
    }

    private DatabricksConfig createSdkConfig() {
        String host = System.getenv("DATABRICKS_HOST");
        String clientId = System.getenv("DATABRICKS_U2M_CLIENT_ID");
        String clientSecret = System.getenv("DATABRICKS_U2M_CLIENT_SECRET");

        DatabricksConfig cfg = new DatabricksConfig()
                .setHost(host.startsWith("https://") ? host : "https://" + host)
                .setAuthType("external-browser")
                .setClientId(clientId)
                .setClientSecret(clientSecret)
                .setScopes(Arrays.asList("all-apis", "offline_access"));

        cfg.resolve();
        return cfg;
    }

    private String getToken() {
        Map<String, String> headers = sdkConfig.authenticate();
        return headers.get("Authorization").substring("Bearer ".length());
    }

    private Connection createConnection() throws Exception {
        String host = System.getenv("DATABRICKS_HOST")
                .replaceFirst("^https://", "").replaceFirst("^http://", "");
        String httpPath = System.getenv("DATABRICKS_HTTP_PATH");
        String token = getToken();

        String url = "jdbc:databricks://" + host + ":443"
            + ";httpPath=" + httpPath
            + ";AuthMech=11;Auth_Flow=0;Auth_AccessToken=" + token
            + ";UserAgentEntry=YourCompany_YourProduct/1.0.0";

        return DriverManager.getConnection(url);
    }

    public Connection getConnection() throws Exception {
        if (System.currentTimeMillis() > (tokenExpiresAt - REFRESH_BUFFER_MS)) {
            // Token nearing expiration, refresh and reconnect
            if (jdbcConnection != null && !jdbcConnection.isClosed()) {
                jdbcConnection.close();
            }
            this.sdkConfig = createSdkConfig();
            this.jdbcConnection = createConnection();
            this.tokenExpiresAt = System.currentTimeMillis() + (55 * 60 * 1000);
        }
        return jdbcConnection;
    }

    public void close() throws Exception {
        if (jdbcConnection != null && !jdbcConnection.isClosed()) {
            jdbcConnection.close();
        }
    }
}

Refresh Token Scope

For applications that need to refresh without user interaction, request offline_access scope:

config.setScopes(Arrays.asList("all-apis", "offline_access"));

Important: The offline_access scope must be enabled for your OAuth app in App connections.

Best Practices

1. Use connection pooling with refresh: Combine with HikariCP for production (see Connection Pooling section)
2. Check token before queries: Avoid mid-query failures
3. Close old connections: Prevent connection leaks during refresh
4. Handle SQLException gracefully: Token expiration may manifest as SQL errors

---

Environment Variables Reference

Variable	Used By	Description
DATABRICKS_HOST	All	Workspace URL
DATABRICKS_HTTP_PATH	All	SQL warehouse HTTP path (e.g., /sql/1.0/warehouses/abc123)
DATABRICKS_WAREHOUSE_ID	All	Alternative to HTTP_PATH; path derived as /sql/1.0/warehouses/<id>
DATABRICKS_TOKEN	PAT, U2M token-env	Personal access token or pre-obtained OAuth token
DATABRICKS_CLIENT_ID	OAuth M2M	Service principal UUID
DATABRICKS_CLIENT_SECRET	OAuth M2M	Service principal OAuth secret
DATABRICKS_U2M_CLIENT_ID	U2M Custom OAuth	Custom OAuth app client ID
DATABRICKS_U2M_CLIENT_SECRET	U2M Custom OAuth	Custom OAuth app secret (if confidential)
DATABRICKS_REDIRECT_URI	U2M Custom OAuth	Redirect URI (default: http://localhost:8080/callback)
DATABRICKS_AUTH_TYPE	Multi-auth binary	Auth type selector

---

JDBC URL Parameters Reference

Timeout Configuration

Parameter	Default	Description
ConnectTimeout	60	Connection timeout in seconds
SocketTimeout	0	Socket read timeout in seconds (0 = infinite)
QueryTimeout	0	Default query timeout in seconds (0 = infinite)
LoginTimeout	60	Login timeout in seconds
AsyncExecPollInterval	200	Polling interval for async queries in milliseconds

SSL/TLS Configuration

Parameter	Default	Description
SSL	1	Enable SSL/TLS (1=enabled, 0=disabled). Always use SSL=1 for Databricks
SSLTrustStore	(system)	Path to custom truststore file for certificate validation
SSLTrustStorePassword	(none)	Password for custom truststore
AllowSelfSignedCerts	0	Allow self-signed certificates (1=allow, 0=reject). Not recommended for production

Warehouse Startup Configuration

Parameter	Default	Description
EnableAutoResume	1	Auto-resume stopped warehouses (1=enabled, 0=disabled)

Example with Timeout/SSL Parameters

String url = "jdbc:databricks://" + host + ":443"
    + ";httpPath=" + httpPath
    + ";AuthMech=11;Auth_Flow=1"
    + ";OAuth2ClientId=" + clientId
    + ";OAuth2Secret=" + clientSecret
    + ";SSL=1"
    + ";ConnectTimeout=30"
    + ";SocketTimeout=300"
    + ";QueryTimeout=600"
    + ";EnableAutoResume=1"
    + ";UserAgentEntry=YourCompany_YourProduct/1.0.0";

Statement-Level Timeout

For per-query timeout control, use Statement.setQueryTimeout():

try (Statement stmt = conn.createStatement()) {
    stmt.setQueryTimeout(300); // 5 minutes
    try (ResultSet rs = stmt.executeQuery("SELECT * FROM large_table")) {
        // Process results
    }
}

---

Required Imports / Maven Dependencies

<!-- Databricks JDBC Driver -->
<dependency>
    <groupId>com.databricks</groupId>
    <artifactId>databricks-jdbc</artifactId>
    <version>3.1.1</version>
</dependency>

<!-- Databricks SDK (for U2M browser flows) -->
<dependency>
    <groupId>com.databricks</groupId>
    <artifactId>databricks-sdk-java</artifactId>
    <version>0.54.0</version>
</dependency>

<!-- HikariCP for connection pooling (recommended for production) -->
<dependency>
    <groupId>com.zaxxer</groupId>
    <artifactId>HikariCP</artifactId>
    <version>5.1.0</version>
</dependency>

<!-- Suppress SDK logging (optional) -->
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-nop</artifactId>
    <version>2.0.16</version>
</dependency>

---

Connection Pooling with HikariCP

Production JDBC applications should use connection pooling for better performance and resource management. HikariCP is the recommended pool for Databricks JDBC.

Basic HikariCP Setup (OAuth M2M)

import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.Statement;

public class JdbcConnectionPoolExample {
    private static HikariDataSource dataSource;

    public static void main(String[] args) throws Exception {
        initializePool();
        try {
            executeQuery();
        } finally {
            closePool();
        }
    }

    private static void initializePool() {
        String host = System.getenv("DATABRICKS_HOST")
                .replaceFirst("^https://", "").replaceFirst("^http://", "");
        String httpPath = System.getenv("DATABRICKS_HTTP_PATH");
        String clientId = System.getenv("DATABRICKS_CLIENT_ID");
        String clientSecret = System.getenv("DATABRICKS_CLIENT_SECRET");

        String jdbcUrl = "jdbc:databricks://" + host + ":443"
            + ";httpPath=" + httpPath
            + ";AuthMech=11;Auth_Flow=1"
            + ";OAuth2ClientId=" + clientId
            + ";OAuth2Secret=" + clientSecret
            + ";SSL=1"
            + ";ConnectTimeout=30"
            + ";SocketTimeout=300"
            + ";EnableAutoResume=1"
            + ";UserAgentEntry=YourCompany_YourProduct/1.0.0";

        HikariConfig config = new HikariConfig();
        config.setJdbcUrl(jdbcUrl);
        config.setPoolName("DatabricksPool");

        // Pool sizing
        config.setMinimumIdle(2);
        config.setMaximumPoolSize(10);

        // Connection lifecycle
        config.setIdleTimeout(300000);        // 5 minutes idle before eviction
        config.setMaxLifetime(1800000);       // 30 minutes max connection lifetime
        config.setConnectionTimeout(30000);   // 30 seconds to get connection from pool
        config.setKeepaliveTime(60000);       // 1 minute keepalive interval

        // Validation
        config.setConnectionTestQuery("SELECT 1");

        dataSource = new HikariDataSource(config);
    }

    private static void executeQuery() throws Exception {
        try (Connection conn = dataSource.getConnection();
             Statement stmt = conn.createStatement();
             ResultSet rs = stmt.executeQuery("SELECT count(*) as cnt FROM samples.nyctaxi.trips")) {
            if (rs.next()) {
                System.out.println("Row count: " + rs.getLong("cnt"));
            }
        }
    }

    private static void closePool() {
        if (dataSource != null && !dataSource.isClosed()) {
            dataSource.close();
        }
    }
}

HikariCP with U2M Token Refresh

For U2M flows where tokens expire, implement a custom connection provider that refreshes tokens:

import com.databricks.sdk.core.DatabricksConfig;
import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
import java.util.Arrays;
import java.util.Map;
import java.util.concurrent.atomic.AtomicReference;

public class JdbcU2MPoolExample {
    private final AtomicReference<HikariDataSource> dataSourceRef = new AtomicReference<>();
    private final AtomicReference<Long> tokenExpiresAt = new AtomicReference<>(0L);
    private static final long REFRESH_BUFFER_MS = 5 * 60 * 1000; // 5 minutes

    public synchronized HikariDataSource getDataSource() {
        long now = System.currentTimeMillis();
        if (now > (tokenExpiresAt.get() - REFRESH_BUFFER_MS)) {
            // Token expiring soon, create new pool with fresh token
            HikariDataSource oldDs = dataSourceRef.get();

            String token = obtainFreshToken();
            HikariDataSource newDs = createPoolWithToken(token);

            dataSourceRef.set(newDs);
            tokenExpiresAt.set(now + (55 * 60 * 1000)); // ~55 minutes

            // Close old pool gracefully
            if (oldDs != null && !oldDs.isClosed()) {
                oldDs.close();
            }
        }
        return dataSourceRef.get();
    }

    private String obtainFreshToken() {
        String host = System.getenv("DATABRICKS_HOST");
        String clientId = System.getenv("DATABRICKS_U2M_CLIENT_ID");
        String clientSecret = System.getenv("DATABRICKS_U2M_CLIENT_SECRET");

        DatabricksConfig config = new DatabricksConfig()
                .setHost(host.startsWith("https://") ? host : "https://" + host)
                .setAuthType("external-browser")
                .setClientId(clientId)
                .setClientSecret(clientSecret)
                .setScopes(Arrays.asList("all-apis", "offline_access"));

        config.resolve();
        Map<String, String> headers = config.authenticate();
        return headers.get("Authorization").substring("Bearer ".length());
    }

    private HikariDataSource createPoolWithToken(String token) {
        String host = System.getenv("DATABRICKS_HOST")
                .replaceFirst("^https://", "").replaceFirst("^http://", "");
        String httpPath = System.getenv("DATABRICKS_HTTP_PATH");

        String jdbcUrl = "jdbc:databricks://" + host + ":443"
            + ";httpPath=" + httpPath
            + ";AuthMech=11;Auth_Flow=0;Auth_AccessToken=" + token
            + ";SSL=1;ConnectTimeout=30;SocketTimeout=300"
            + ";UserAgentEntry=YourCompany_YourProduct/1.0.0";

        HikariConfig config = new HikariConfig();
        config.setJdbcUrl(jdbcUrl);
        config.setPoolName("DatabricksU2MPool");
        config.setMinimumIdle(2);
        config.setMaximumPoolSize(10);
        config.setMaxLifetime(1800000);  // 30 minutes (less than token TTL)
        config.setConnectionTestQuery("SELECT 1");

        return new HikariDataSource(config);
    }

    public void shutdown() {
        HikariDataSource ds = dataSourceRef.get();
        if (ds != null && !ds.isClosed()) {
            ds.close();
        }
    }
}

Pool Configuration Best Practices

Setting	Recommended Value	Notes
minimumIdle	2-5	Keep a few connections warm
maximumPoolSize	10-20	Based on concurrent query needs
idleTimeout	300000 (5 min)	Evict idle connections
maxLifetime	1800000 (30 min)	Less than token TTL for U2M
connectionTimeout	30000 (30 sec)	Time to wait for connection
connectionTestQuery	SELECT 1	Validates connection before use
leakDetectionThreshold	60000 (1 min)	Detect leaked connections

Important for U2M: Set maxLifetime shorter than the token TTL (~1 hour) to ensure connections don't outlive their tokens.

Pool Monitoring

Monitor pool health at runtime using HikariCP's MXBean:

// Get pool statistics
HikariPoolMXBean poolMXBean = dataSource.getHikariPoolMXBean();
System.out.println("Active connections: " + poolMXBean.getActiveConnections());
System.out.println("Idle connections: " + poolMXBean.getIdleConnections());
System.out.println("Total connections: " + poolMXBean.getTotalConnections());
System.out.println("Threads awaiting connection: " + poolMXBean.getThreadsAwaitingConnection());

Health check pattern:

public boolean isPoolHealthy() {
    HikariPoolMXBean mxBean = dataSource.getHikariPoolMXBean();
    return !dataSource.isClosed() 
        && mxBean.getTotalConnections() > 0
        && mxBean.getThreadsAwaitingConnection() < mxBean.getTotalConnections();
}

---

Multi-Auth Pattern

Support multiple auth types in a single binary using DATABRICKS_AUTH_TYPE:

String authType = System.getenv("DATABRICKS_AUTH_TYPE");
if (authType == null || authType.isBlank()) authType = "oauth_m2m";

String host = System.getenv("DATABRICKS_HOST").replaceFirst("^https://", "").replaceFirst("^http://", "");
String httpPath = System.getenv("DATABRICKS_HTTP_PATH");
String url;

switch (authType.toLowerCase()) {
    case "pat":
        String token = System.getenv("DATABRICKS_TOKEN");
        url = "jdbc:databricks://" + host + ":443;httpPath=" + httpPath
            + ";AuthMech=3;UID=token;PWD=" + token
            + ";UserAgentEntry=YourCompany/1.0.0";
        break;
    case "oauth_m2m":
        String clientId = System.getenv("DATABRICKS_CLIENT_ID");
        String clientSecret = System.getenv("DATABRICKS_CLIENT_SECRET");
        url = "jdbc:databricks://" + host + ":443;httpPath=" + httpPath
            + ";AuthMech=11;Auth_Flow=1;OAuth2ClientId=" + clientId
            + ";OAuth2Secret=" + clientSecret
            + ";UserAgentEntry=YourCompany/1.0.0";
        break;
    case "u2m_custom_oauth_app":
        String customToken = getTokenViaCustomOAuthApp(); // see helper below
        url = "jdbc:databricks://" + host + ":443;httpPath=" + httpPath
            + ";AuthMech=11;Auth_Flow=0;Auth_AccessToken=" + customToken
            + ";UserAgentEntry=YourCompany/1.0.0";
        break;
    case "u2m_token_env":
        String envToken = System.getenv("DATABRICKS_ACCESS_TOKEN");
        if (envToken == null || envToken.isBlank()) envToken = System.getenv("DATABRICKS_TOKEN");
        url = "jdbc:databricks://" + host + ":443;httpPath=" + httpPath
            + ";AuthMech=11;Auth_Flow=0;Auth_AccessToken=" + envToken
            + ";UserAgentEntry=YourCompany/1.0.0";
        break;
    default:
        throw new IllegalArgumentException("Unknown DATABRICKS_AUTH_TYPE: " + authType);
}

Helper for U2M Custom OAuth App token acquisition:

private static String getTokenViaCustomOAuthApp() {
    String host = System.getenv("DATABRICKS_HOST");
    String clientId = System.getenv("DATABRICKS_U2M_CLIENT_ID");
    String clientSecret = System.getenv("DATABRICKS_U2M_CLIENT_SECRET");
    String redirectUri = System.getenv("DATABRICKS_REDIRECT_URI");
    
    DatabricksConfig config = new DatabricksConfig()
            .setHost(host.startsWith("https://") ? host : "https://" + host)
            .setAuthType("external-browser")
            .setClientId(clientId)
            .setClientSecret(clientSecret)  // Required by Java SDK
            .setScopes(Arrays.asList("all-apis"));
    
    if (redirectUri != null && !redirectUri.isBlank()) {
        config.setOAuthRedirectUrl(redirectUri);
    }
    config.resolve();
    
    Map<String, String> headers = config.authenticate();
    return headers.get("Authorization").substring("Bearer ".length());
}

DATABRICKS_AUTH_TYPE	JDBC Auth	Token Source
oauth_m2m (default)	Auth_Flow=1	Driver handles
pat	AuthMech=3	DATABRICKS_TOKEN
u2m_custom_oauth_app	Auth_Flow=0	SDK custom app
u2m_token_env	Auth_Flow=0	DATABRICKS_ACCESS_TOKEN

---

Implementation Learnings

Key findings from testing that differ from documentation:

1. Java SDK Requires Client Secret for U2M

When using the Java SDK to obtain tokens for JDBC (U2M custom OAuth app), you must provide DATABRICKS_U2M_CLIENT_SECRET:

DatabricksConfig config = new DatabricksConfig()
        .setHost(host)
        .setAuthType("external-browser")
        .setClientId(System.getenv("DATABRICKS_U2M_CLIENT_ID"))
        .setClientSecret(System.getenv("DATABRICKS_U2M_CLIENT_SECRET"))  // Required!
        .setScopes(Arrays.asList("all-apis"));

Without client secret: "cannot configure default credentials"

2. Token Extraction Pattern

Extract the token from SDK authentication headers:

config.resolve();
Map<String, String> headers = config.authenticate();  // Opens browser
String token = headers.get("Authorization").substring("Bearer ".length());

3. Host Normalization Differs

• SDK expects https:// prefix: config.setHost("https://workspace.cloud.databricks.com")
• JDBC expects bare hostname: jdbc:databricks://workspace.cloud.databricks.com:443

Always normalize the host appropriately for each:

// For SDK
String sdkHost = host.startsWith("https://") ? host : "https://" + host;

// For JDBC URL
String jdbcHost = host.replaceFirst("^https://", "").replaceFirst("^http://", "");

---

SDK Workarounds Summary (Java SDK v0.54.0)

Workaround	Code	Without It
Set scopes	.setScopes(Arrays.asList("all-apis"))	NullPointerException at TokenCacheUtils
Call resolve	config.resolve() before config.authenticate()	NullPointerException at Consent.<init>
Set client secret	.setClientSecret(...)	"cannot configure default credentials"
Set redirect URL (if custom)	.setOAuthRedirectUrl(...)	Must match App connections registration

---

Redirect URI Reference

Scenario	Redirect URI	Notes
Custom OAuth app (Java SDK default)	http://localhost:8080/callback	Java SDK default
Custom OAuth app (configured)	Via DATABRICKS_REDIRECT_URI	Must match App connections registration
JDBC Auth_Flow=2 (driver-native)	http://localhost:8020 (hardcoded)	Not recommended; doesn't support custom apps

---

Validation Query

Verify User-Agent telemetry is being recorded correctly:

SELECT event_time, user_agent, action_name, request_params
FROM system.access.audit
WHERE event_time > current_timestamp() - INTERVAL 1 HOUR
  AND lower(user_agent) LIKE '%yourcompany%'
ORDER BY event_time DESC
LIMIT 10;

---

Auth Isolation

Run tests with a clean environment using env -i. Important: Include MAVEN_OPTS for Java 17+.

# Set MAVEN_OPTS for Java 17+
export MAVEN_OPTS="--add-opens=java.base/java.nio=ALL-UNNAMED"

# PAT test
env -i HOME="$HOME" PATH="$PATH" JAVA_HOME="$JAVA_HOME" MAVEN_OPTS="$MAVEN_OPTS" \
  DATABRICKS_HOST="$DATABRICKS_HOST" \
  DATABRICKS_HTTP_PATH="$DATABRICKS_HTTP_PATH" \
  DATABRICKS_TOKEN="$DATABRICKS_TOKEN" \
  DATABRICKS_AUTH_TYPE=pat \
  mvn -q exec:exec -Dexec.mainClass="com.example.JdbcPatExample"

# OAuth M2M test
env -i HOME="$HOME" PATH="$PATH" JAVA_HOME="$JAVA_HOME" MAVEN_OPTS="$MAVEN_OPTS" \
  DATABRICKS_HOST="$DATABRICKS_HOST" \
  DATABRICKS_HTTP_PATH="$DATABRICKS_HTTP_PATH" \
  DATABRICKS_CLIENT_ID="$DATABRICKS_CLIENT_ID" \
  DATABRICKS_CLIENT_SECRET="$DATABRICKS_CLIENT_SECRET" \
  DATABRICKS_AUTH_TYPE=oauth_m2m \
  mvn -q exec:exec -Dexec.mainClass="com.example.JdbcOAuthM2MExample"

# U2M Custom OAuth App test (needs client secret for Java SDK)
env -i HOME="$HOME" PATH="$PATH" JAVA_HOME="$JAVA_HOME" MAVEN_OPTS="$MAVEN_OPTS" \
  DATABRICKS_HOST="$DATABRICKS_HOST" \
  DATABRICKS_HTTP_PATH="$DATABRICKS_HTTP_PATH" \
  DATABRICKS_U2M_CLIENT_ID="$DATABRICKS_U2M_CLIENT_ID" \
  DATABRICKS_U2M_CLIENT_SECRET="$DATABRICKS_U2M_CLIENT_SECRET" \
  DATABRICKS_REDIRECT_URI="$DATABRICKS_REDIRECT_URI" \
  DATABRICKS_AUTH_TYPE=u2m_custom_oauth_app \
  mvn -q exec:exec -Dexec.mainClass="com.example.JdbcU2MCustomOAuthAppExample"

# U2M Token-Env test
env -i HOME="$HOME" PATH="$PATH" JAVA_HOME="$JAVA_HOME" MAVEN_OPTS="$MAVEN_OPTS" \
  DATABRICKS_HOST="$DATABRICKS_HOST" \
  DATABRICKS_HTTP_PATH="$DATABRICKS_HTTP_PATH" \
  DATABRICKS_ACCESS_TOKEN="$DATABRICKS_ACCESS_TOKEN" \
  DATABRICKS_AUTH_TYPE=u2m_token_env \
  mvn -q exec:exec -Dexec.mainClass="com.example.JdbcU2MTokenEnvExample"

---

Java 17+

The OSS driver 3.x uses Arrow. On Java 17+, the JVM must allow Arrow's reflection via --add-opens=java.base/java.nio=ALL-UNNAMED.

⚠️ CRITICAL: Use exec:exec, NOT exec:java in exec-maven-plugin

exec:java runs code inside the Maven JVM process — jvmArguments are completely ignored. --add-opens flags never take effect and Apache Arrow fails at startup with:

java.lang.ExceptionInInitializerError: Failed to initialize MemoryUtil
InaccessibleObjectException: Unable to make field accessible...

Only exec:exec forks a real child JVM where startup flags work. The <classpath/> element (no attributes) is a Maven magic marker that expands to the full classpath.

Correct pom.xml configuration (exec:exec with --add-opens):

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>exec-maven-plugin</artifactId>
    <version>3.1.0</version>
    <configuration>
        <executable>java</executable>
        <arguments>
            <argument>--add-opens=java.base/java.nio=ALL-UNNAMED</argument>
            <argument>-classpath</argument>
            <classpath/>  <!-- Maven expands this to the full classpath string -->
            <argument>com.example.YourMainClass</argument>
        </arguments>
    </configuration>
</plugin>

Run with: mvn exec:exec (not mvn exec:java)

Via MAVEN_OPTS (alternative, works with both exec:java and exec:exec):

export MAVEN_OPTS="--add-opens=java.base/java.nio=ALL-UNNAMED"

---

Error Handling Patterns

Production JDBC applications should handle connection and query errors gracefully.

Basic Error Handling

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;

public class JdbcErrorHandlingExample {
    public static void main(String[] args) {
        try {
            executeQuery();
        } catch (SQLException e) {
            handleSQLException(e);
        } catch (Exception e) {
            handleGeneralError(e);
        }
    }

    private static void executeQuery() throws Exception {
        String host = System.getenv("DATABRICKS_HOST");
        String httpPath = System.getenv("DATABRICKS_HTTP_PATH");
        String clientId = System.getenv("DATABRICKS_CLIENT_ID");
        String clientSecret = System.getenv("DATABRICKS_CLIENT_SECRET");

        // Validate required environment variables
        if (host == null || host.isBlank()) {
            throw new IllegalStateException("DATABRICKS_HOST is required");
        }
        if (httpPath == null || httpPath.isBlank()) {
            throw new IllegalStateException("DATABRICKS_HTTP_PATH is required");
        }

        host = host.replaceFirst("^https://", "").replaceFirst("^http://", "");
        String url = "jdbc:databricks://" + host + ":443"
            + ";httpPath=" + httpPath
            + ";AuthMech=11;Auth_Flow=1"
            + ";OAuth2ClientId=" + clientId
            + ";OAuth2Secret=" + clientSecret
            + ";UserAgentEntry=YourCompany_YourProduct/1.0.0";

        try (Connection conn = DriverManager.getConnection(url);
             Statement stmt = conn.createStatement();
             ResultSet rs = stmt.executeQuery("SELECT 1 as test")) {
            if (rs.next()) {
                System.out.println("Success: " + rs.getInt("test"));
            }
        }
    }

    private static void handleSQLException(SQLException e) {
        String sqlState = e.getSQLState();
        int errorCode = e.getErrorCode();
        String message = e.getMessage();

        System.err.println("SQL Error [" + sqlState + "] (" + errorCode + "): " + message);

        if (message.contains("invalid_client") || message.contains("Invalid credentials")) {
            System.err.println("Authentication failed:");
            System.err.println("  - Verify DATABRICKS_CLIENT_ID is correct");
            System.err.println("  - Verify DATABRICKS_CLIENT_SECRET is correct");
        } else if (message.contains("INVALID_ACCESS_TOKEN") || message.contains("Token expired")) {
            System.err.println("Access token expired:");
            System.err.println("  - U2M tokens expire after ~1 hour");
            System.err.println("  - Re-authenticate or implement token refresh");
        } else if (message.contains("Could not open client transport")) {
            System.err.println("Connection failed:");
            System.err.println("  - Verify DATABRICKS_HOST is correct");
            System.err.println("  - Check network connectivity and firewall rules");
        } else if (message.contains("HTTP Response code: 503")) {
            System.err.println("Warehouse unavailable - may be starting up");
            System.err.println("  - Wait and retry with backoff");
        } else if ("08S01".equals(sqlState)) {
            System.err.println("Communication link failure - network issue or timeout");
        }
    }

    private static void handleGeneralError(Exception e) {
        String message = e.getMessage();

        if (e instanceof java.net.UnknownHostException) {
            System.err.println("Cannot resolve host - check DATABRICKS_HOST");
        } else if (e instanceof java.net.SocketTimeoutException) {
            System.err.println("Connection timeout - check network/firewall");
        } else {
            System.err.println("Error: " + message);
            e.printStackTrace();
        }
    }
}

Retry Pattern with Exponential Backoff

import java.sql.Connection;
import java.sql.SQLException;

public class JdbcRetryHelper {
    private static final int MAX_RETRIES = 3;
    private static final long INITIAL_BACKOFF_MS = 2000;

    @FunctionalInterface
    public interface JdbcOperation<T> {
        T execute(Connection conn) throws SQLException;
    }

    public static <T> T withRetry(Connection conn, JdbcOperation<T> operation) throws SQLException {
        int attempt = 0;
        SQLException lastException = null;

        while (attempt <= MAX_RETRIES) {
            try {
                return operation.execute(conn);
            } catch (SQLException e) {
                lastException = e;
                if (!isRetryable(e) || attempt >= MAX_RETRIES) {
                    throw e;
                }
                long backoff = INITIAL_BACKOFF_MS * (long) Math.pow(2, attempt);
                System.err.println("Retry " + (attempt + 1) + "/" + MAX_RETRIES 
                    + " after " + backoff + "ms: " + e.getMessage());
                try {
                    Thread.sleep(backoff);
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    throw new SQLException("Interrupted during retry", ie);
                }
                attempt++;
            }
        }
        throw lastException;
    }

    private static boolean isRetryable(SQLException e) {
        String msg = e.getMessage();
        String sqlState = e.getSQLState();
        return msg.contains("503")           // Service unavailable
            || msg.contains("429")           // Rate limited
            || msg.contains("TEMPORARILY_UNAVAILABLE")
            || "08S01".equals(sqlState)      // Communication link failure
            || "08003".equals(sqlState);     // Connection not open
    }
}

// Usage:
// String result = JdbcRetryHelper.withRetry(conn, c -> {
//     try (Statement stmt = c.createStatement();
//          ResultSet rs = stmt.executeQuery("SELECT count(*) FROM table")) {
//         rs.next();
//         return rs.getString(1);
//     }
// });

---

SQL State Reference

Common SQL States returned by the Databricks JDBC driver:

SQL State	Meaning	Retryable	Remediation
08S01	Communication link failure	Yes	Network issue; retry with backoff
08003	Connection not open	Yes	Connection was closed; retry
08006	Connection failure	Yes	Network issue; retry with backoff
22000	Data exception	No	Check query/data types
42000	Syntax error or access violation	No	Fix SQL or check permissions
HY000	General error	Depends	Check error message for details

Retryable HTTP Status Codes:

• 503 - Service unavailable (warehouse starting up)
• 429 - Rate limited (too many requests)
• 500 - Internal server error (transient)

---

JDBC URL Parameters Reference

Complete list of commonly used Databricks JDBC URL parameters:

Parameter	Required	Description
httpPath	Yes	SQL warehouse HTTP path (e.g., /sql/1.0/warehouses/<id>)
AuthMech	Yes	Auth mechanism: 3 (PAT), 11 (OAuth)
Auth_Flow	For OAuth	0 (token pass-through), 1 (M2M client credentials)
UID	For PAT	Set to token
PWD	For PAT	Personal access token value
OAuth2ClientId	For M2M	Service principal client ID
OAuth2Secret	For M2M	Service principal client secret
Auth_AccessToken	For Auth_Flow=0	Pre-obtained access token
SSL	Recommended	1 to enable SSL (always use in production)
ConnectTimeout	Recommended	Connection timeout in seconds (default: 60)
SocketTimeout	Recommended	Socket read timeout in seconds (default: 600)
EnableAutoResume	Optional	1 to auto-resume stopped warehouses
UserAgentEntry	PWAF Required	ISV telemetry string (format: ISV_Product/version)

Example URL:

jdbc:databricks://workspace.cloud.databricks.com:443
  ;httpPath=/sql/1.0/warehouses/abc123
  ;AuthMech=11;Auth_Flow=1
  ;OAuth2ClientId=<client-id>
  ;OAuth2Secret=<client-secret>
  ;SSL=1;ConnectTimeout=30;SocketTimeout=300
  ;UserAgentEntry=YourCompany_YourProduct/1.0.0

---

Troubleshooting

Common Errors

Error	Cause	Solution
more than one authorization method configured	Multiple auth env vars set	Use env -i to pass only vars for one auth type
Connection timeout	Network/firewall issue	Verify host is accessible, increase ConnectTimeout
Invalid credentials	Wrong token/secret	Verify credentials
Could not open client transport	Network blocked	Check firewall allows port 443
HTTP Response code: 503	Warehouse starting	Wait and retry, or enable EnableAutoResume=1

OAuth-Specific Errors

Error	Cause	Solution
OAuth application with client_id not available	Using M2M client_id for U2M	Use DATABRICKS_U2M_CLIENT_ID (custom OAuth app)
redirect_uri mismatch	URI doesn't match registration	Verify URI in App connections matches DATABRICKS_REDIRECT_URI
invalid_client	Wrong client ID/secret	Verify credentials match OAuth app
invalid_grant	Auth code expired/reused	Auth codes are single-use; restart flow

SDK Errors

Error	Cause	Solution
NullPointerException at TokenCacheUtils	Scopes not set	.setScopes(Arrays.asList("all-apis"))
NullPointerException at Consent.<init>	HTTP client not initialized	config.resolve() before config.authenticate()
cannot configure default credentials	Custom OAuth app missing client secret	Add DATABRICKS_U2M_CLIENT_SECRET for custom OAuth apps
Token expired	Access token TTL (~1 hour)	Re-run browser flow or use refresh_token

Java 17+ Issues

Error	Cause	Solution
InaccessibleObjectException	Arrow reflection on Java 17+	Add --add-opens=java.base/java.nio=ALL-UNNAMED

---

Key Differences from Databricks SDK for Java

Aspect	Databricks JDBC Driver	Databricks SDK for Java
Purpose	SQL queries via warehouse	REST APIs (UC, Jobs, Clusters)
SQL Warehouse	Required (httpPath)	Not needed
Auth Config	JDBC URL parameters (AuthMech, Auth_Flow)	DatabricksConfig{} setters
User-Agent	UserAgentEntry property (per-connection)	UserAgent.withProduct() (global)
Token Refresh	Driver handles M2M refresh	SDK handles M2M refresh
Java 17+	Needs --add-opens for Arrow	No special flags needed