| name | databricks-isv-java-jdbc |
| description | PWAF-compliant Databricks JDBC Driver (databricks-jdbc v3+): PAT, OAuth M2M, OAuth U2M (custom OAuth app, token-env). Use when building or testing Java SQL warehouse integrations. |
Databricks JDBC Driver (ISV)
Use this skill when implementing or testing Databricks JDBC Driver (databricks-jdbc v3+) connectivity for PWAF-compliant SQL warehouse integrations.
PWAF Documentation Links
Requirements
- Driver:
com.databricks:databricks-jdbc:3.1.1 (or later 3.x) from Maven Central. OSS driver only.
- SDK:
com.databricks:databricks-sdk-java:0.54.0 – Required for U2M browser-based token acquisition. Not needed for PAT/M2M/token-env.
- Java: 11+ required. Java 17+ needs
--add-opens=java.base/java.nio=ALL-UNNAMED.
- User-Agent (required): Set
UserAgentEntry on every connection URL (format <isv>_<product>/<version>).
Authentication Decision Guide
Which authentication method to use?
Production / automated workloads?
→ OAuth M2M (Auth_Flow=1) ✅ RECOMMENDED
User-interactive (ISV custom OAuth app)?
→ U2M Custom OAuth App (SDK → Auth_Flow=0) ✅ SUPPORTED
Already have an OAuth access token?
→ U2M Token-Env (Auth_Flow=0) ✅ SUPPORTED
Local development/testing only?
→ PAT (AuthMech=3) ⚠️ LIMITED
Note: These examples do NOT use the SDK's built-in databricks-cli OAuth app. ISV/partner applications should register custom OAuth apps in App connections for proper branding, audit trails, and scoped permissions.
Authentication Comparison
| Method | PWAF Status | JDBC Auth | Token Source | Browser |
|---|
| PAT | ⚠️ Limited | AuthMech=3 | User provides | No |
| OAuth M2M | ✅ Recommended | Auth_Flow=1 | Driver handles | No |
| U2M Custom OAuth App | ✅ Supported | Auth_Flow=0 | Java SDK (custom app) | Yes |
| U2M Token-Env | ✅ Supported | Auth_Flow=0 | Pre-obtained from env | No |
Auth Mapping (JDBC Parameters)
| Auth | JDBC Parameters | Token Source |
|---|
| PAT | AuthMech=3, UID=token, PWD=<token> | User provides PAT directly |
| OAuth M2M | AuthMech=11, Auth_Flow=1, OAuth2ClientId, OAuth2Secret | JDBC driver does client-credentials |
| OAuth U2M (custom app) | AuthMech=11, Auth_Flow=0, Auth_AccessToken | Java SDK external-browser (custom OAuth app) |
| OAuth U2M (token-env) | AuthMech=11, Auth_Flow=0, Auth_AccessToken | Pre-obtained from env variable |
Both U2M flows use the same JDBC parameters (Auth_Flow=0 token pass-through). The difference is how the token is obtained.
CLIENT_ID Distinction (CRITICAL)
| Variable | Purpose | Used By |
|---|
DATABRICKS_CLIENT_ID | M2M service principal UUID | OAuth M2M only |
DATABRICKS_U2M_CLIENT_ID | Custom OAuth app client ID | U2M custom OAuth app |
Using the wrong client_id causes: "OAuth application with client_id not available in Databricks account"
Why SDK + Auth_Flow=0 Instead of JDBC Auth_Flow=2?
The JDBC driver's native Auth_Flow=2 uses a built-in OAuth app with a hardcoded redirect port (8020). It does not support custom OAuth apps or custom redirect URIs.
For ISV integrations that need custom OAuth apps, the proven pattern is:
- Use the Databricks Java SDK (
DatabricksConfig with authType="external-browser") to obtain the token
- Pass the token to the JDBC driver via
Auth_Flow=0 (Auth_AccessToken)
This approach supports custom OAuth apps with configurable redirect URIs, which is required for proper ISV branding and audit trails.
Compute / Target
- SQL warehouse only: Connection uses
httpPath (e.g., /sql/1.0/warehouses/<id>). The JDBC driver does not support jobs compute.
- Config: Require either
DATABRICKS_HTTP_PATH or DATABRICKS_WAREHOUSE_ID (path derived as /sql/1.0/warehouses/<id>).
User-Agent (Required per PWAF)
Set on every JDBC connection URL:
String url = "jdbc:databricks://" + host + ":443"
+ ";httpPath=" + httpPath
+ ";AuthMech=3;UID=token;PWD=" + token
+ ";UserAgentEntry=YourCompany_YourProduct/1.0.0";
For SDK operations, also register static telemetry:
import com.databricks.sdk.core.UserAgent;
UserAgent.withProduct("YourCompany_YourProduct", "1.0.0");
UserAgent.withPartner("YourCompany");
Environment Variables Reference
| Variable | Required For | Description |
|---|
DATABRICKS_HOST | All | Workspace URL (e.g., https://myworkspace.cloud.databricks.com) |
DATABRICKS_HTTP_PATH | All | SQL warehouse HTTP path (e.g., /sql/1.0/warehouses/abc123) |
DATABRICKS_WAREHOUSE_ID | Alternative | Warehouse ID (path derived as /sql/1.0/warehouses/<id>) |
DATABRICKS_TOKEN | PAT | Personal access token |
DATABRICKS_CLIENT_ID | OAuth M2M | Service principal UUID |
DATABRICKS_CLIENT_SECRET | OAuth M2M | Service principal OAuth secret |
DATABRICKS_U2M_CLIENT_ID | U2M Custom OAuth | Custom OAuth app client ID from App connections |
DATABRICKS_U2M_CLIENT_SECRET | U2M Custom OAuth | Custom OAuth app client secret (Java SDK requires this) |
DATABRICKS_REDIRECT_URI | U2M Custom OAuth (optional) | Custom redirect URI (default: http://localhost:8080/callback) |
DATABRICKS_ACCESS_TOKEN | U2M Token-Env | Pre-obtained OAuth access token |
DATABRICKS_AUTH_TYPE | Multi-auth | Auth type selector: pat, oauth_m2m, u2m_custom_oauth_app, u2m_token_env |
Important: Do not mix M2M and U2M environment variables. Use env -i for clean test environments.
Complete Examples
PAT Authentication (Testing Only)
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;
public class JdbcPatExample {
public static void main(String[] args) throws Exception {
String host = System.getenv("DATABRICKS_HOST");
String token = System.getenv("DATABRICKS_TOKEN");
String httpPath = System.getenv("DATABRICKS_HTTP_PATH");
host = host.replaceFirst("^https://", "").replaceFirst("^http://", "");
String url = "jdbc:databricks://" + host + ":443"
+ ";httpPath=" + httpPath
+ ";AuthMech=3"
+ ";UID=token"
+ ";PWD=" + token
+ ";UserAgentEntry=YourCompany_YourProduct/1.0.0";
try (Connection conn = DriverManager.getConnection(url);
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery("SELECT 1 as test")) {
if (rs.next()) {
System.out.println("PAT OK: " + rs.getInt("test"));
}
}
}
}
Env vars: DATABRICKS_HOST, DATABRICKS_TOKEN, DATABRICKS_HTTP_PATH
OAuth M2M Authentication (Production Recommended)
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;
public class JdbcOAuthM2MExample {
public static void main(String[] args) throws Exception {
String host = System.getenv("DATABRICKS_HOST");
String clientId = System.getenv("DATABRICKS_CLIENT_ID");
String clientSecret = System.getenv("DATABRICKS_CLIENT_SECRET");
String httpPath = System.getenv("DATABRICKS_HTTP_PATH");
host = host.replaceFirst("^https://", "").replaceFirst("^http://", "");
String url = "jdbc:databricks://" + host + ":443"
+ ";httpPath=" + httpPath
+ ";AuthMech=11"
+ ";Auth_Flow=1"
+ ";OAuth2ClientId=" + clientId
+ ";OAuth2Secret=" + clientSecret
+ ";UserAgentEntry=YourCompany_YourProduct/1.0.0";
try (Connection conn = DriverManager.getConnection(url);
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery("SELECT 1 as test")) {
if (rs.next()) {
System.out.println("OAuth M2M OK: " + rs.getInt("test"));
}
}
}
}
Env vars: DATABRICKS_HOST, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET, DATABRICKS_HTTP_PATH
Setup: Create service principal in Account Console → Settings → Service principals. Generate OAuth secret.
U2M Custom OAuth App (SDK → JDBC)
Uses a custom OAuth app registered in App connections:
import com.databricks.sdk.core.DatabricksConfig;
import com.databricks.sdk.core.UserAgent;
import java.sql.Connection;
import java.sql.DriverManager;
import java.util.Arrays;
import java.util.Map;
public class JdbcU2MCustomOAuthAppExample {
public static void main(String[] args) throws Exception {
String host = System.getenv("DATABRICKS_HOST");
String clientId = System.getenv("DATABRICKS_U2M_CLIENT_ID");
String clientSecret = System.getenv("DATABRICKS_U2M_CLIENT_SECRET");
String redirectUri = System.getenv("DATABRICKS_REDIRECT_URI");
String httpPath = System.getenv("DATABRICKS_HTTP_PATH");
UserAgent.withProduct("YourCompany_YourProduct", "1.0.0");
UserAgent.withPartner("YourCompany");
String normalizedHost = host.startsWith("https://") ? host : "https://" + host;
DatabricksConfig config = new DatabricksConfig()
.setHost(normalizedHost)
.setAuthType("external-browser")
.setClientId(clientId)
.setScopes(Arrays.asList("all-apis"));
if (clientSecret != null && !clientSecret.isBlank()) {
config.setClientSecret(clientSecret);
}
if (redirectUri != null && !redirectUri.isBlank()) {
config.setOAuthRedirectUrl(redirectUri);
}
config.resolve();
Map<String, String> headers = config.authenticate();
String token = headers.get("Authorization").substring("Bearer ".length());
String hostNorm = host.replaceFirst("^https://", "").replaceFirst("^http://", "");
String url = "jdbc:databricks://" + hostNorm + ":443"
+ ";httpPath=" + httpPath
+ ";AuthMech=11;Auth_Flow=0;Auth_AccessToken=" + token
+ ";UserAgentEntry=YourCompany_YourProduct/1.0.0";
try (Connection conn = DriverManager.getConnection(url)) {
System.out.println("U2M Custom OAuth App OK");
}
}
}
Env vars: DATABRICKS_HOST, DATABRICKS_U2M_CLIENT_ID, DATABRICKS_HTTP_PATH
Recommended: DATABRICKS_U2M_CLIENT_SECRET (required by Java SDK for proper auth configuration)
Optional: DATABRICKS_REDIRECT_URI (must match App connections registration)
CRITICAL: Use DATABRICKS_U2M_CLIENT_ID for custom OAuth apps — NOT DATABRICKS_CLIENT_ID (that's for M2M).
Note: Testing showed the Java SDK requires DATABRICKS_U2M_CLIENT_SECRET for custom OAuth apps to properly configure external-browser auth. Without it, you may get "cannot configure default credentials" errors.
U2M Token-Env (Pre-obtained Token)
For headless/CI environments with a pre-obtained token:
import java.sql.Connection;
import java.sql.DriverManager;
public class JdbcU2MTokenEnvExample {
public static void main(String[] args) throws Exception {
String host = System.getenv("DATABRICKS_HOST");
String httpPath = System.getenv("DATABRICKS_HTTP_PATH");
String accessToken = System.getenv("DATABRICKS_ACCESS_TOKEN");
if (accessToken == null || accessToken.isBlank()) {
accessToken = System.getenv("DATABRICKS_TOKEN");
}
String hostNorm = host.replaceFirst("^https://", "").replaceFirst("^http://", "");
String url = "jdbc:databricks://" + hostNorm + ":443"
+ ";httpPath=" + httpPath
+ ";AuthMech=11"
+ ";Auth_Flow=0"
+ ";Auth_AccessToken=" + accessToken
+ ";UserAgentEntry=YourCompany_YourProduct/1.0.0";
try (Connection conn = DriverManager.getConnection(url)) {
System.out.println("U2M Token-Env OK");
}
}
}
Env vars: DATABRICKS_HOST, DATABRICKS_HTTP_PATH, DATABRICKS_ACCESS_TOKEN (or DATABRICKS_TOKEN)
Token Refresh for U2M Flows
U2M access tokens expire after approximately 1 hour. For long-running JDBC applications, you need to handle token refresh and reconnection.
Token Lifetime Summary
| Auth Type | Token TTL | Refresh Strategy |
|---|
| PAT | User-configured (90 days default) | Generate new token manually |
| OAuth M2M | ~1 hour | JDBC driver handles automatically |
| U2M Custom OAuth App | ~1 hour | Re-authenticate via SDK, reconnect JDBC |
| U2M Token-Env | ~1 hour | Application must handle |
JDBC Token Refresh Pattern
Since JDBC connections use a token at connection time, you need to obtain a fresh token and create a new connection when tokens expire:
public class JdbcTokenRefreshExample {
private DatabricksConfig sdkConfig;
private Connection jdbcConnection;
private long tokenExpiresAt;
private static final long REFRESH_BUFFER_MS = 5 * 60 * 1000;
public JdbcTokenRefreshExample() throws Exception {
this.sdkConfig = createSdkConfig();
this.jdbcConnection = createConnection();
this.tokenExpiresAt = System.currentTimeMillis() + (55 * 60 * 1000);
}
private DatabricksConfig createSdkConfig() {
String host = System.getenv("DATABRICKS_HOST");
String clientId = System.getenv("DATABRICKS_U2M_CLIENT_ID");
String clientSecret = System.getenv("DATABRICKS_U2M_CLIENT_SECRET");
DatabricksConfig cfg = new DatabricksConfig()
.setHost(host.startsWith("https://") ? host : "https://" + host)
.setAuthType("external-browser")
.setClientId(clientId)
.setClientSecret(clientSecret)
.setScopes(Arrays.asList("all-apis", "offline_access"));
cfg.resolve();
return cfg;
}
private String getToken() {
Map<String, String> headers = sdkConfig.authenticate();
return headers.get("Authorization").substring("Bearer ".length());
}
private Connection createConnection() throws Exception {
String host = System.getenv("DATABRICKS_HOST")
.replaceFirst("^https://", "").replaceFirst("^http://", "");
String httpPath = System.getenv("DATABRICKS_HTTP_PATH");
String token = getToken();
String url = "jdbc:databricks://" + host + ":443"
+ ";httpPath=" + httpPath
+ ";AuthMech=11;Auth_Flow=0;Auth_AccessToken=" + token
+ ";UserAgentEntry=YourCompany_YourProduct/1.0.0";
return DriverManager.getConnection(url);
}
public Connection getConnection() throws Exception {
if (System.currentTimeMillis() > (tokenExpiresAt - REFRESH_BUFFER_MS)) {
if (jdbcConnection != null && !jdbcConnection.isClosed()) {
jdbcConnection.close();
}
this.sdkConfig = createSdkConfig();
this.jdbcConnection = createConnection();
this.tokenExpiresAt = System.currentTimeMillis() + (55 * 60 * 1000);
}
return jdbcConnection;
}
public void close() throws Exception {
if (jdbcConnection != null && !jdbcConnection.isClosed()) {
jdbcConnection.close();
}
}
}
Refresh Token Scope
For applications that need to refresh without user interaction, request offline_access scope:
config.setScopes(Arrays.asList("all-apis", "offline_access"));
Important: The offline_access scope must be enabled for your OAuth app in App connections.
Best Practices
- Use connection pooling with refresh: Combine with HikariCP for production (see Connection Pooling section)
- Check token before queries: Avoid mid-query failures
- Close old connections: Prevent connection leaks during refresh
- Handle SQLException gracefully: Token expiration may manifest as SQL errors
Environment Variables Reference
| Variable | Used By | Description |
|---|
DATABRICKS_HOST | All | Workspace URL |
DATABRICKS_HTTP_PATH | All | SQL warehouse HTTP path (e.g., /sql/1.0/warehouses/abc123) |
DATABRICKS_WAREHOUSE_ID | All | Alternative to HTTP_PATH; path derived as /sql/1.0/warehouses/<id> |
DATABRICKS_TOKEN | PAT, U2M token-env | Personal access token or pre-obtained OAuth token |
DATABRICKS_CLIENT_ID | OAuth M2M | Service principal UUID |
DATABRICKS_CLIENT_SECRET | OAuth M2M | Service principal OAuth secret |
DATABRICKS_U2M_CLIENT_ID | U2M Custom OAuth | Custom OAuth app client ID |
DATABRICKS_U2M_CLIENT_SECRET | U2M Custom OAuth | Custom OAuth app secret (if confidential) |
DATABRICKS_REDIRECT_URI | U2M Custom OAuth | Redirect URI (default: http://localhost:8080/callback) |
DATABRICKS_AUTH_TYPE | Multi-auth binary | Auth type selector |
JDBC URL Parameters Reference
Timeout Configuration
| Parameter | Default | Description |
|---|
ConnectTimeout | 60 | Connection timeout in seconds |
SocketTimeout | 0 | Socket read timeout in seconds (0 = infinite) |
QueryTimeout | 0 | Default query timeout in seconds (0 = infinite) |
LoginTimeout | 60 | Login timeout in seconds |
AsyncExecPollInterval | 200 | Polling interval for async queries in milliseconds |
SSL/TLS Configuration
| Parameter | Default | Description |
|---|
SSL | 1 | Enable SSL/TLS (1=enabled, 0=disabled). Always use SSL=1 for Databricks |
SSLTrustStore | (system) | Path to custom truststore file for certificate validation |
SSLTrustStorePassword | (none) | Password for custom truststore |
AllowSelfSignedCerts | 0 | Allow self-signed certificates (1=allow, 0=reject). Not recommended for production |
Warehouse Startup Configuration
| Parameter | Default | Description |
|---|
EnableAutoResume | 1 | Auto-resume stopped warehouses (1=enabled, 0=disabled) |
Example with Timeout/SSL Parameters
String url = "jdbc:databricks://" + host + ":443"
+ ";httpPath=" + httpPath
+ ";AuthMech=11;Auth_Flow=1"
+ ";OAuth2ClientId=" + clientId
+ ";OAuth2Secret=" + clientSecret
+ ";SSL=1"
+ ";ConnectTimeout=30"
+ ";SocketTimeout=300"
+ ";QueryTimeout=600"
+ ";EnableAutoResume=1"
+ ";UserAgentEntry=YourCompany_YourProduct/1.0.0";
Statement-Level Timeout
For per-query timeout control, use Statement.setQueryTimeout():
try (Statement stmt = conn.createStatement()) {
stmt.setQueryTimeout(300);
try (ResultSet rs = stmt.executeQuery("SELECT * FROM large_table")) {
}
}
Required Imports / Maven Dependencies
<dependency>
<groupId>com.databricks</groupId>
<artifactId>databricks-jdbc</artifactId>
<version>3.1.1</version>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>databricks-sdk-java</artifactId>
<version>0.54.0</version>
</dependency>
<dependency>
<groupId>com.zaxxer</groupId>
<artifactId>HikariCP</artifactId>
<version>5.1.0</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-nop</artifactId>
<version>2.0.16</version>
</dependency>
Connection Pooling with HikariCP
Production JDBC applications should use connection pooling for better performance and resource management. HikariCP is the recommended pool for Databricks JDBC.
Basic HikariCP Setup (OAuth M2M)
import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.Statement;
public class JdbcConnectionPoolExample {
private static HikariDataSource dataSource;
public static void main(String[] args) throws Exception {
initializePool();
try {
executeQuery();
} finally {
closePool();
}
}
private static void initializePool() {
String host = System.getenv("DATABRICKS_HOST")
.replaceFirst("^https://", "").replaceFirst("^http://", "");
String httpPath = System.getenv("DATABRICKS_HTTP_PATH");
String clientId = System.getenv("DATABRICKS_CLIENT_ID");
String clientSecret = System.getenv("DATABRICKS_CLIENT_SECRET");
String jdbcUrl = "jdbc:databricks://" + host + ":443"
+ ";httpPath=" + httpPath
+ ";AuthMech=11;Auth_Flow=1"
+ ";OAuth2ClientId=" + clientId
+ ";OAuth2Secret=" + clientSecret
+ ";SSL=1"
+ ";ConnectTimeout=30"
+ ";SocketTimeout=300"
+ ";EnableAutoResume=1"
+ ";UserAgentEntry=YourCompany_YourProduct/1.0.0";
HikariConfig config = new HikariConfig();
config.setJdbcUrl(jdbcUrl);
config.setPoolName("DatabricksPool");
config.setMinimumIdle(2);
config.setMaximumPoolSize(10);
config.setIdleTimeout(300000);
config.setMaxLifetime(1800000);
config.setConnectionTimeout(30000);
config.setKeepaliveTime(60000);
config.setConnectionTestQuery("SELECT 1");
dataSource = new HikariDataSource(config);
}
private static void executeQuery() throws Exception {
try (Connection conn = dataSource.getConnection();
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery("SELECT count(*) as cnt FROM samples.nyctaxi.trips")) {
if (rs.next()) {
System.out.println("Row count: " + rs.getLong("cnt"));
}
}
}
private static void closePool() {
if (dataSource != null && !dataSource.isClosed()) {
dataSource.close();
}
}
}
HikariCP with U2M Token Refresh
For U2M flows where tokens expire, implement a custom connection provider that refreshes tokens:
import com.databricks.sdk.core.DatabricksConfig;
import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
import java.util.Arrays;
import java.util.Map;
import java.util.concurrent.atomic.AtomicReference;
public class JdbcU2MPoolExample {
private final AtomicReference<HikariDataSource> dataSourceRef = new AtomicReference<>();
private final AtomicReference<Long> tokenExpiresAt = new AtomicReference<>(0L);
private static final long REFRESH_BUFFER_MS = 5 * 60 * 1000;
public synchronized HikariDataSource getDataSource() {
long now = System.currentTimeMillis();
if (now > (tokenExpiresAt.get() - REFRESH_BUFFER_MS)) {
HikariDataSource oldDs = dataSourceRef.get();
String token = obtainFreshToken();
HikariDataSource newDs = createPoolWithToken(token);
dataSourceRef.set(newDs);
tokenExpiresAt.set(now + (55 * 60 * 1000));
if (oldDs != null && !oldDs.isClosed()) {
oldDs.close();
}
}
return dataSourceRef.get();
}
private String obtainFreshToken() {
String host = System.getenv("DATABRICKS_HOST");
String clientId = System.getenv("DATABRICKS_U2M_CLIENT_ID");
String clientSecret = System.getenv("DATABRICKS_U2M_CLIENT_SECRET");
DatabricksConfig config = new DatabricksConfig()
.setHost(host.startsWith("https://") ? host : "https://" + host)
.setAuthType("external-browser")
.setClientId(clientId)
.setClientSecret(clientSecret)
.setScopes(Arrays.asList("all-apis", "offline_access"));
config.resolve();
Map<String, String> headers = config.authenticate();
return headers.get("Authorization").substring("Bearer ".length());
}
private HikariDataSource createPoolWithToken(String token) {
String host = System.getenv("DATABRICKS_HOST")
.replaceFirst("^https://", "").replaceFirst("^http://", "");
String httpPath = System.getenv("DATABRICKS_HTTP_PATH");
String jdbcUrl = "jdbc:databricks://" + host + ":443"
+ ";httpPath=" + httpPath
+ ";AuthMech=11;Auth_Flow=0;Auth_AccessToken=" + token
+ ";SSL=1;ConnectTimeout=30;SocketTimeout=300"
+ ";UserAgentEntry=YourCompany_YourProduct/1.0.0";
HikariConfig config = new HikariConfig();
config.setJdbcUrl(jdbcUrl);
config.setPoolName("DatabricksU2MPool");
config.setMinimumIdle(2);
config.setMaximumPoolSize(10);
config.setMaxLifetime(1800000);
config.setConnectionTestQuery("SELECT 1");
return new HikariDataSource(config);
}
public void shutdown() {
HikariDataSource ds = dataSourceRef.get();
if (ds != null && !ds.isClosed()) {
ds.close();
}
}
}
Pool Configuration Best Practices
| Setting | Recommended Value | Notes |
|---|
minimumIdle | 2-5 | Keep a few connections warm |
maximumPoolSize | 10-20 | Based on concurrent query needs |
idleTimeout | 300000 (5 min) | Evict idle connections |
maxLifetime | 1800000 (30 min) | Less than token TTL for U2M |
connectionTimeout | 30000 (30 sec) | Time to wait for connection |
connectionTestQuery | SELECT 1 | Validates connection before use |
leakDetectionThreshold | 60000 (1 min) | Detect leaked connections |
Important for U2M: Set maxLifetime shorter than the token TTL (~1 hour) to ensure connections don't outlive their tokens.
Pool Monitoring
Monitor pool health at runtime using HikariCP's MXBean:
HikariPoolMXBean poolMXBean = dataSource.getHikariPoolMXBean();
System.out.println("Active connections: " + poolMXBean.getActiveConnections());
System.out.println("Idle connections: " + poolMXBean.getIdleConnections());
System.out.println("Total connections: " + poolMXBean.getTotalConnections());
System.out.println("Threads awaiting connection: " + poolMXBean.getThreadsAwaitingConnection());
Health check pattern:
public boolean isPoolHealthy() {
HikariPoolMXBean mxBean = dataSource.getHikariPoolMXBean();
return !dataSource.isClosed()
&& mxBean.getTotalConnections() > 0
&& mxBean.getThreadsAwaitingConnection() < mxBean.getTotalConnections();
}
Multi-Auth Pattern
Support multiple auth types in a single binary using DATABRICKS_AUTH_TYPE:
String authType = System.getenv("DATABRICKS_AUTH_TYPE");
if (authType == null || authType.isBlank()) authType = "oauth_m2m";
String host = System.getenv("DATABRICKS_HOST").replaceFirst("^https://", "").replaceFirst("^http://", "");
String httpPath = System.getenv("DATABRICKS_HTTP_PATH");
String url;
switch (authType.toLowerCase()) {
case "pat":
String token = System.getenv("DATABRICKS_TOKEN");
url = "jdbc:databricks://" + host + ":443;httpPath=" + httpPath
+ ";AuthMech=3;UID=token;PWD=" + token
+ ";UserAgentEntry=YourCompany/1.0.0";
break;
case "oauth_m2m":
String clientId = System.getenv("DATABRICKS_CLIENT_ID");
String clientSecret = System.getenv("DATABRICKS_CLIENT_SECRET");
url = "jdbc:databricks://" + host + ":443;httpPath=" + httpPath
+ ";AuthMech=11;Auth_Flow=1;OAuth2ClientId=" + clientId
+ ";OAuth2Secret=" + clientSecret
+ ";UserAgentEntry=YourCompany/1.0.0";
break;
case "u2m_custom_oauth_app":
String customToken = getTokenViaCustomOAuthApp();
url = "jdbc:databricks://" + host + ":443;httpPath=" + httpPath
+ ";AuthMech=11;Auth_Flow=0;Auth_AccessToken=" + customToken
+ ";UserAgentEntry=YourCompany/1.0.0";
break;
case "u2m_token_env":
String envToken = System.getenv("DATABRICKS_ACCESS_TOKEN");
if (envToken == null || envToken.isBlank()) envToken = System.getenv("DATABRICKS_TOKEN");
url = "jdbc:databricks://" + host + ":443;httpPath=" + httpPath
+ ";AuthMech=11;Auth_Flow=0;Auth_AccessToken=" + envToken
+ ";UserAgentEntry=YourCompany/1.0.0";
break;
default:
throw new IllegalArgumentException("Unknown DATABRICKS_AUTH_TYPE: " + authType);
}
Helper for U2M Custom OAuth App token acquisition:
private static String getTokenViaCustomOAuthApp() {
String host = System.getenv("DATABRICKS_HOST");
String clientId = System.getenv("DATABRICKS_U2M_CLIENT_ID");
String clientSecret = System.getenv("DATABRICKS_U2M_CLIENT_SECRET");
String redirectUri = System.getenv("DATABRICKS_REDIRECT_URI");
DatabricksConfig config = new DatabricksConfig()
.setHost(host.startsWith("https://") ? host : "https://" + host)
.setAuthType("external-browser")
.setClientId(clientId)
.setClientSecret(clientSecret)
.setScopes(Arrays.asList("all-apis"));
if (redirectUri != null && !redirectUri.isBlank()) {
config.setOAuthRedirectUrl(redirectUri);
}
config.resolve();
Map<String, String> headers = config.authenticate();
return headers.get("Authorization").substring("Bearer ".length());
}
DATABRICKS_AUTH_TYPE | JDBC Auth | Token Source |
|---|
oauth_m2m (default) | Auth_Flow=1 | Driver handles |
pat | AuthMech=3 | DATABRICKS_TOKEN |
u2m_custom_oauth_app | Auth_Flow=0 | SDK custom app |
u2m_token_env | Auth_Flow=0 | DATABRICKS_ACCESS_TOKEN |
Implementation Learnings
Key findings from testing that differ from documentation:
1. Java SDK Requires Client Secret for U2M
When using the Java SDK to obtain tokens for JDBC (U2M custom OAuth app), you must provide DATABRICKS_U2M_CLIENT_SECRET:
DatabricksConfig config = new DatabricksConfig()
.setHost(host)
.setAuthType("external-browser")
.setClientId(System.getenv("DATABRICKS_U2M_CLIENT_ID"))
.setClientSecret(System.getenv("DATABRICKS_U2M_CLIENT_SECRET"))
.setScopes(Arrays.asList("all-apis"));
Without client secret: "cannot configure default credentials"
2. Token Extraction Pattern
Extract the token from SDK authentication headers:
config.resolve();
Map<String, String> headers = config.authenticate();
String token = headers.get("Authorization").substring("Bearer ".length());
3. Host Normalization Differs
- SDK expects
https:// prefix: config.setHost("https://workspace.cloud.databricks.com")
- JDBC expects bare hostname:
jdbc:databricks://workspace.cloud.databricks.com:443
Always normalize the host appropriately for each:
String sdkHost = host.startsWith("https://") ? host : "https://" + host;
String jdbcHost = host.replaceFirst("^https://", "").replaceFirst("^http://", "");
SDK Workarounds Summary (Java SDK v0.54.0)
| Workaround | Code | Without It |
|---|
| Set scopes | .setScopes(Arrays.asList("all-apis")) | NullPointerException at TokenCacheUtils |
| Call resolve | config.resolve() before config.authenticate() | NullPointerException at Consent.<init> |
| Set client secret | .setClientSecret(...) | "cannot configure default credentials" |
| Set redirect URL (if custom) | .setOAuthRedirectUrl(...) | Must match App connections registration |
Redirect URI Reference
| Scenario | Redirect URI | Notes |
|---|
| Custom OAuth app (Java SDK default) | http://localhost:8080/callback | Java SDK default |
| Custom OAuth app (configured) | Via DATABRICKS_REDIRECT_URI | Must match App connections registration |
| JDBC Auth_Flow=2 (driver-native) | http://localhost:8020 (hardcoded) | Not recommended; doesn't support custom apps |
Validation Query
Verify User-Agent telemetry is being recorded correctly:
SELECT event_time, user_agent, action_name, request_params
FROM system.access.audit
WHERE event_time > current_timestamp() - INTERVAL 1 HOUR
AND lower(user_agent) LIKE '%yourcompany%'
ORDER BY event_time DESC
LIMIT 10;
Auth Isolation
Run tests with a clean environment using env -i. Important: Include MAVEN_OPTS for Java 17+.
export MAVEN_OPTS="--add-opens=java.base/java.nio=ALL-UNNAMED"
env -i HOME="$HOME" PATH="$PATH" JAVA_HOME="$JAVA_HOME" MAVEN_OPTS="$MAVEN_OPTS" \
DATABRICKS_HOST="$DATABRICKS_HOST" \
DATABRICKS_HTTP_PATH="$DATABRICKS_HTTP_PATH" \
DATABRICKS_TOKEN="$DATABRICKS_TOKEN" \
DATABRICKS_AUTH_TYPE=pat \
mvn -q exec:exec -Dexec.mainClass="com.example.JdbcPatExample"
env -i HOME="$HOME" PATH="$PATH" JAVA_HOME="$JAVA_HOME" MAVEN_OPTS="$MAVEN_OPTS" \
DATABRICKS_HOST="$DATABRICKS_HOST" \
DATABRICKS_HTTP_PATH="$DATABRICKS_HTTP_PATH" \
DATABRICKS_CLIENT_ID="$DATABRICKS_CLIENT_ID" \
DATABRICKS_CLIENT_SECRET="$DATABRICKS_CLIENT_SECRET" \
DATABRICKS_AUTH_TYPE=oauth_m2m \
mvn -q exec:exec -Dexec.mainClass="com.example.JdbcOAuthM2MExample"
env -i HOME="$HOME" PATH="$PATH" JAVA_HOME="$JAVA_HOME" MAVEN_OPTS="$MAVEN_OPTS" \
DATABRICKS_HOST="$DATABRICKS_HOST" \
DATABRICKS_HTTP_PATH="$DATABRICKS_HTTP_PATH" \
DATABRICKS_U2M_CLIENT_ID="$DATABRICKS_U2M_CLIENT_ID" \
DATABRICKS_U2M_CLIENT_SECRET="$DATABRICKS_U2M_CLIENT_SECRET" \
DATABRICKS_REDIRECT_URI="$DATABRICKS_REDIRECT_URI" \
DATABRICKS_AUTH_TYPE=u2m_custom_oauth_app \
mvn -q exec:exec -Dexec.mainClass="com.example.JdbcU2MCustomOAuthAppExample"
env -i HOME="$HOME" PATH="$PATH" JAVA_HOME="$JAVA_HOME" MAVEN_OPTS="$MAVEN_OPTS" \
DATABRICKS_HOST="$DATABRICKS_HOST" \
DATABRICKS_HTTP_PATH="$DATABRICKS_HTTP_PATH" \
DATABRICKS_ACCESS_TOKEN="$DATABRICKS_ACCESS_TOKEN" \
DATABRICKS_AUTH_TYPE=u2m_token_env \
mvn -q exec:exec -Dexec.mainClass="com.example.JdbcU2MTokenEnvExample"
Java 17+
The OSS driver 3.x uses Arrow. On Java 17+, the JVM must allow Arrow's reflection via --add-opens=java.base/java.nio=ALL-UNNAMED.
⚠️ CRITICAL: Use exec:exec, NOT exec:java in exec-maven-plugin
exec:java runs code inside the Maven JVM process — jvmArguments are completely ignored. --add-opens flags never take effect and Apache Arrow fails at startup with:
java.lang.ExceptionInInitializerError: Failed to initialize MemoryUtil
InaccessibleObjectException: Unable to make field accessible...
Only exec:exec forks a real child JVM where startup flags work. The <classpath/> element (no attributes) is a Maven magic marker that expands to the full classpath.
Correct pom.xml configuration (exec:exec with --add-opens):
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>3.1.0</version>
<configuration>
<executable>java</executable>
<arguments>
<argument>--add-opens=java.base/java.nio=ALL-UNNAMED</argument>
<argument>-classpath</argument>
<classpath/>
<argument>com.example.YourMainClass</argument>
</arguments>
</configuration>
</plugin>
Run with: mvn exec:exec (not mvn exec:java)
Via MAVEN_OPTS (alternative, works with both exec:java and exec:exec):
export MAVEN_OPTS="--add-opens=java.base/java.nio=ALL-UNNAMED"
Error Handling Patterns
Production JDBC applications should handle connection and query errors gracefully.
Basic Error Handling
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
public class JdbcErrorHandlingExample {
public static void main(String[] args) {
try {
executeQuery();
} catch (SQLException e) {
handleSQLException(e);
} catch (Exception e) {
handleGeneralError(e);
}
}
private static void executeQuery() throws Exception {
String host = System.getenv("DATABRICKS_HOST");
String httpPath = System.getenv("DATABRICKS_HTTP_PATH");
String clientId = System.getenv("DATABRICKS_CLIENT_ID");
String clientSecret = System.getenv("DATABRICKS_CLIENT_SECRET");
if (host == null || host.isBlank()) {
throw new IllegalStateException("DATABRICKS_HOST is required");
}
if (httpPath == null || httpPath.isBlank()) {
throw new IllegalStateException("DATABRICKS_HTTP_PATH is required");
}
host = host.replaceFirst("^https://", "").replaceFirst("^http://", "");
String url = "jdbc:databricks://" + host + ":443"
+ ";httpPath=" + httpPath
+ ";AuthMech=11;Auth_Flow=1"
+ ";OAuth2ClientId=" + clientId
+ ";OAuth2Secret=" + clientSecret
+ ";UserAgentEntry=YourCompany_YourProduct/1.0.0";
try (Connection conn = DriverManager.getConnection(url);
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery("SELECT 1 as test")) {
if (rs.next()) {
System.out.println("Success: " + rs.getInt("test"));
}
}
}
private static void handleSQLException(SQLException e) {
String sqlState = e.getSQLState();
int errorCode = e.getErrorCode();
String message = e.getMessage();
System.err.println("SQL Error [" + sqlState + "] (" + errorCode + "): " + message);
if (message.contains("invalid_client") || message.contains("Invalid credentials")) {
System.err.println("Authentication failed:");
System.err.println(" - Verify DATABRICKS_CLIENT_ID is correct");
System.err.println(" - Verify DATABRICKS_CLIENT_SECRET is correct");
} else if (message.contains("INVALID_ACCESS_TOKEN") || message.contains("Token expired")) {
System.err.println("Access token expired:");
System.err.println(" - U2M tokens expire after ~1 hour");
System.err.println(" - Re-authenticate or implement token refresh");
} else if (message.contains("Could not open client transport")) {
System.err.println("Connection failed:");
System.err.println(" - Verify DATABRICKS_HOST is correct");
System.err.println(" - Check network connectivity and firewall rules");
} else if (message.contains("HTTP Response code: 503")) {
System.err.println("Warehouse unavailable - may be starting up");
System.err.println(" - Wait and retry with backoff");
} else if ("08S01".equals(sqlState)) {
System.err.println("Communication link failure - network issue or timeout");
}
}
private static void handleGeneralError(Exception e) {
String message = e.getMessage();
if (e instanceof java.net.UnknownHostException) {
System.err.println("Cannot resolve host - check DATABRICKS_HOST");
} else if (e instanceof java.net.SocketTimeoutException) {
System.err.println("Connection timeout - check network/firewall");
} else {
System.err.println("Error: " + message);
e.printStackTrace();
}
}
}
Retry Pattern with Exponential Backoff
import java.sql.Connection;
import java.sql.SQLException;
public class JdbcRetryHelper {
private static final int MAX_RETRIES = 3;
private static final long INITIAL_BACKOFF_MS = 2000;
@FunctionalInterface
public interface JdbcOperation<T> {
T execute(Connection conn) throws SQLException;
}
public static <T> T withRetry(Connection conn, JdbcOperation<T> operation) throws SQLException {
int attempt = 0;
SQLException lastException = null;
while (attempt <= MAX_RETRIES) {
try {
return operation.execute(conn);
} catch (SQLException e) {
lastException = e;
if (!isRetryable(e) || attempt >= MAX_RETRIES) {
throw e;
}
long backoff = INITIAL_BACKOFF_MS * (long) Math.pow(2, attempt);
System.err.println("Retry " + (attempt + 1) + "/" + MAX_RETRIES
+ " after " + backoff + "ms: " + e.getMessage());
try {
Thread.sleep(backoff);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
throw new SQLException("Interrupted during retry", ie);
}
attempt++;
}
}
throw lastException;
}
private static boolean isRetryable(SQLException e) {
String msg = e.getMessage();
String sqlState = e.getSQLState();
return msg.contains("503")
|| msg.contains("429")
|| msg.contains("TEMPORARILY_UNAVAILABLE")
|| "08S01".equals(sqlState)
|| "08003".equals(sqlState);
}
}
SQL State Reference
Common SQL States returned by the Databricks JDBC driver:
| SQL State | Meaning | Retryable | Remediation |
|---|
08S01 | Communication link failure | Yes | Network issue; retry with backoff |
08003 | Connection not open | Yes | Connection was closed; retry |
08006 | Connection failure | Yes | Network issue; retry with backoff |
22000 | Data exception | No | Check query/data types |
42000 | Syntax error or access violation | No | Fix SQL or check permissions |
HY000 | General error | Depends | Check error message for details |
Retryable HTTP Status Codes:
503 - Service unavailable (warehouse starting up)429 - Rate limited (too many requests)500 - Internal server error (transient)
JDBC URL Parameters Reference
Complete list of commonly used Databricks JDBC URL parameters:
| Parameter | Required | Description |
|---|
httpPath | Yes | SQL warehouse HTTP path (e.g., /sql/1.0/warehouses/<id>) |
AuthMech | Yes | Auth mechanism: 3 (PAT), 11 (OAuth) |
Auth_Flow | For OAuth | 0 (token pass-through), 1 (M2M client credentials) |
UID | For PAT | Set to token |
PWD | For PAT | Personal access token value |
OAuth2ClientId | For M2M | Service principal client ID |
OAuth2Secret | For M2M | Service principal client secret |
Auth_AccessToken | For Auth_Flow=0 | Pre-obtained access token |
SSL | Recommended | 1 to enable SSL (always use in production) |
ConnectTimeout | Recommended | Connection timeout in seconds (default: 60) |
SocketTimeout | Recommended | Socket read timeout in seconds (default: 600) |
EnableAutoResume | Optional | 1 to auto-resume stopped warehouses |
UserAgentEntry | PWAF Required | ISV telemetry string (format: ISV_Product/version) |
Example URL:
jdbc:databricks://workspace.cloud.databricks.com:443
;httpPath=/sql/1.0/warehouses/abc123
;AuthMech=11;Auth_Flow=1
;OAuth2ClientId=<client-id>
;OAuth2Secret=<client-secret>
;SSL=1;ConnectTimeout=30;SocketTimeout=300
;UserAgentEntry=YourCompany_YourProduct/1.0.0
Troubleshooting
Common Errors
| Error | Cause | Solution |
|---|
more than one authorization method configured | Multiple auth env vars set | Use env -i to pass only vars for one auth type |
| Connection timeout | Network/firewall issue | Verify host is accessible, increase ConnectTimeout |
Invalid credentials | Wrong token/secret | Verify credentials |
Could not open client transport | Network blocked | Check firewall allows port 443 |
HTTP Response code: 503 | Warehouse starting | Wait and retry, or enable EnableAutoResume=1 |
OAuth-Specific Errors
| Error | Cause | Solution |
|---|
OAuth application with client_id not available | Using M2M client_id for U2M | Use DATABRICKS_U2M_CLIENT_ID (custom OAuth app) |
redirect_uri mismatch | URI doesn't match registration | Verify URI in App connections matches DATABRICKS_REDIRECT_URI |
invalid_client | Wrong client ID/secret | Verify credentials match OAuth app |
invalid_grant | Auth code expired/reused | Auth codes are single-use; restart flow |
SDK Errors
| Error | Cause | Solution |
|---|
NullPointerException at TokenCacheUtils | Scopes not set | .setScopes(Arrays.asList("all-apis")) |
NullPointerException at Consent.<init> | HTTP client not initialized | config.resolve() before config.authenticate() |
cannot configure default credentials | Custom OAuth app missing client secret | Add DATABRICKS_U2M_CLIENT_SECRET for custom OAuth apps |
| Token expired | Access token TTL (~1 hour) | Re-run browser flow or use refresh_token |
Java 17+ Issues
| Error | Cause | Solution |
|---|
InaccessibleObjectException | Arrow reflection on Java 17+ | Add --add-opens=java.base/java.nio=ALL-UNNAMED |
Key Differences from Databricks SDK for Java
| Aspect | Databricks JDBC Driver | Databricks SDK for Java |
|---|
| Purpose | SQL queries via warehouse | REST APIs (UC, Jobs, Clusters) |
| SQL Warehouse | Required (httpPath) | Not needed |
| Auth Config | JDBC URL parameters (AuthMech, Auth_Flow) | DatabricksConfig{} setters |
| User-Agent | UserAgentEntry property (per-connection) | UserAgent.withProduct() (global) |
| Token Refresh | Driver handles M2M refresh | SDK handles M2M refresh |
| Java 17+ | Needs --add-opens for Arrow | No special flags needed |
