matlab-connect-databricks

name	matlab-connect-databricks
description	Connect MATLAB to Databricks via Spark (Databricks Connect) or JDBC (Database Toolbox). Use when setting up the MATLAB Interface for Databricks, configuring authentication (OauthU2M, OauthM2M, PAT), creating Spark sessions with getDatabricksSession(), reading Unity Catalog tables with server-side filtering, creating JDBC connections with databricks.JDBCConnection or StandaloneJDBCConnection, connecting to SQL Warehouses, selecting JDBC drivers (Simba/OSS), or writing data back to Databricks. Triggers on: Databricks Connect, Spark from MATLAB, getDatabricksSession, .databrickscfg, databricks.JDBCConnection, StandaloneJDBCConnection, SQLWarehouse, Databricks JDBC, Databricks cluster, large table server-side filtering.
license	https://www.mathworks.com/content/dam/mathworks/license/pmrl/license.md
metadata	{"author":"MathWorks","version":"1.0"}

Connect MATLAB to Databricks

Connect MATLAB to Databricks via Spark (Databricks Connect) or JDBC (Database Toolbox). This skill covers first-time setup, authentication, path selection, session/connection creation, and data operations through both paths.

When to Use

• First-time setup of the MATLAB Interface for Databricks package
• Configuring .databrickscfg and authentication (OauthU2M, OauthM2M, PAT)
• Choosing between Spark and JDBC for a Databricks workflow
• Reading large tables via Spark with server-side filtering (getDatabricksSession)
• Creating JDBC connections to clusters or SQL Warehouses (databricks.JDBCConnection, SQLWarehouse.connect())
• Standalone JDBC connectivity without the full package (StandaloneJDBCConnection)
• Writing data back to Databricks (via Spark DataFrames or JDBC sqlwrite)
• Selecting and configuring JDBC drivers (Simba vs. OSS)

When NOT to Use

• Generic Database Toolbox operations after connection is established — use matlab-use-database
• ODBC connections (databricks.ODBCConnection)
• Databricks REST APIs (Clusters, Jobs, DBFS, Unity Catalog admin)
• MLflow from MATLAB
• Statement Execution REST API
• Deploying compiled MATLAB code to Databricks clusters (Job workflow)
• DuckDB — use matlab-use-duckdb
• Databricks notebooks, Databricks CLI, or databricks-sdk Python workflows
• PySpark without MATLAB context (pure Python Spark usage)

Decision Framework: Spark vs. JDBC

Scenario	Path	Why
Large table (millions of rows), need server-side filtering before pulling locally	Spark	DataFrame operations run on cluster; only filtered results transfer
SQL queries on small-to-medium datasets	JDBC	Direct SQL via Database Toolbox; simpler setup
Need DataFrame transformations (withColumn, select, filter chains)	Spark	Native DataFrame API; operations stay on cluster
Writing large data with performance optimization	JDBC	Simba driver's UseNativeQuery optimizes sqlwrite
No MATLAB Interface for Databricks package installed	JDBC	StandaloneJDBCConnection works with just Database Toolbox + driver jar
Need to use Database Explorer app	JDBC	saveSource() + copyToken() integration
Reading files from Unity Catalog Volumes (CSV, Parquet, JSON)	Spark	spark.read.format().load() with Volumes paths
Interactive exploration with sqlread/fetch	JDBC	Standard Database Toolbox workflow on j.Connection

Default: Use Spark when the user mentions large data, server-side filtering, or DataFrames. Use JDBC when the user mentions SQL queries, Database Toolbox, or small/medium datasets. If unclear, ask the user about their data size and preferred workflow.

Workflow

0. Obtain the package — Ask if the user has the MATLAB Interface for Databricks. If not, direct them to https://www.mathworks.com/solutions/partners/databricks.html (or use StandaloneJDBCConnection for JDBC-only without the package)
1. Setup — Run setup from the package's Software/MATLAB directory. It configures settings, .databrickscfg, and installs the Databricks Connect library via pip into a venv
2. Startup — Run startup to add package paths (required once per MATLAB session)
3. Choose path — Use the Decision Framework above to select Spark or JDBC
4. Connect — Create a session (getDatabricksSession) or connection (databricks.JDBCConnection)
5. Verify — Spark: table(spark.range(1)). JDBC: check j.Connection.Message is empty
6. Operate — Read, filter, write data using the appropriate path's API
7. Close — JDBC: close(j). Spark: sessions are managed automatically

Key Functions

Spark Path

Function	Purpose
getDatabricksSession()	Creates a Spark session (classic compute)
getDatabricksSession(serverless=true)	Creates a serverless session (no cluster, 10-min timeout)
spark.read().table("catalog.schema.table")	Reads a Unity Catalog table as a DataFrame
spark.read.format(fmt).load(path)	Reads files from Volumes (csv, parquet, json)
DF.filter(expr)	Server-side row filtering
DF.select(cols)	Server-side column selection
DF.limit(n)	Server-side row limiting
DF.withColumn(name, col)	Adds/transforms a column (requires Column objects)
table(DF)	Converts DataFrame to MATLAB table (pulls data locally)
matlab.sparkutils.table2dataset(T, spark)	Converts MATLAB table back to Spark DataFrame
DF.write.mode(m).format(f).saveAsTable(name)	Writes DataFrame to Unity Catalog table
matlab.pyspark.sql.functions.col(name)	Creates a Column reference
matlab.pyspark.sql.functions.lit(value)	Creates a literal Column constant

JDBC Path

Function	Purpose
databricks.JDBCConnection()	Creates a JDBC connection (full package)
StandaloneJDBCConnection()	Creates a JDBC connection (no package dependencies)
databricks.SQLWarehouse.connect()	Connects to a SQL Warehouse by ID
j.Connection	The database.jdbc.connection object for Database Toolbox functions
j.testConnection()	Verifies connection is working
j.saveSource()	Saves connection as a Database Toolbox data source
close(j)	Closes connection and releases resources

Patterns

First-Time Setup

Run setup from the package's Software/MATLAB directory. It is interactive — follow prompts for host URL, auth method, cluster ID, and Databricks Connect library installation.

The Databricks Connect library is downloaded via pip into Software/MATLAB/Connect/<version>/venv/. Requires Python 3.10-3.12. If the download fails repeatedly, ask the user to check with their IT team — do not retry with modified arguments.

cd('/path/to/databricks-package/Software/MATLAB')
setup
startup

After setup, MATLAB's pyenv must point to the venv Python. If getDatabricksSession fails with "databricks.connect package is not installed":

terminate(pyenv);
pyenv(Version="/path/to/databricks-package/Software/MATLAB/Connect/17.3/venv/bin/python");

Warning: If Python is already loaded InProcess, terminate(pyenv) fails. A full MATLAB restart is required. Do not attempt to switch pyenv mid-session after Python has been used.

For authentication configuration details (.databrickscfg format, profiles, environment variables, token caching), see references/authentication.md — consult when configuring auth methods or troubleshooting credential issues.

Spark: Read and Filter a Table

Data stays on the cluster until explicitly collected. Filter server-side first, then collect.

spark = getDatabricksSession(authMethod="PAT");
DF = spark.read().table("catalog.schema.sensor_readings");
filtered = DF.filter("temperature > 100 AND event_date > '2024-01-01'");
T = table(filtered);

Spark: Serverless Session

No cluster needed. Starts instantly with 10-minute inactivity timeout. Requires Python 3.11-3.12 and Databricks Connect >= 15.4.

spark = getDatabricksSession(serverless=true);
DF = spark.read().table("catalog.schema.events");
T = table(DF.limit(50));

Spark: Add Computed Columns

withColumn requires Column objects — not raw scalars. Use col() for references and lit() for constants.

import matlab.pyspark.sql.functions.col
import matlab.pyspark.sql.functions.lit

DF = spark.read().table("catalog.schema.measurements");
DF2 = DF.withColumn("temp_fahrenheit", col("temp_celsius") * lit(9/5) + lit(32));

Spark: Write Back to Databricks

Always specify .mode() — without it, writes fail if the target exists.

DF_new = matlab.sparkutils.table2dataset(T, spark);
DF_new.write.mode("overwrite").format("delta").saveAsTable("catalog.schema.output_table");

Spark: Read Files from Volumes

DF = spark.read.format("csv").option("header", "true").load("/Volumes/catalog/schema/volume/data.csv");
T = table(DF.limit(1000));

JDBC: Cluster Connection

j = databricks.JDBCConnection(catalog="mycatalog", schema="myschema");
data = sqlread(j.Connection, "mytable");
close(j);

JDBC: SQL Warehouse Connection

warehouse.connect() returns a database.jdbc.connection directly — pass it to sqlread/fetch without .Connection.

warehouse = databricks.SQLWarehouse;
warehouse.id = "abc123def456";
conn = warehouse.connect();
data = fetch(conn, "SELECT * FROM mycatalog.myschema.mytable LIMIT 10");
close(conn);

JDBC: Standalone (No Package)

When the user does NOT have the MATLAB Interface for Databricks, use StandaloneJDBCConnection. Requires Database Toolbox and the Simba driver jar only. See references/standalone-jdbc.md for setup and JSON template.

j = StandaloneJDBCConnection(schema="myschema", catalog="mycatalog");
data = fetch(j.Connection, "SELECT * FROM mytable LIMIT 10");
close(j);

JDBC: On-Databricks (Browser MATLAB)

The JDBC driver's OAuth flow cannot open a browser when MATLAB runs on a Databricks cluster. Use package-managed auth instead.

if databricks.internal.isOnDatabricks()
    j = databricks.JDBCConnection(authMethod="OauthU2M", useDriverAuth=false);
else
    j = databricks.JDBCConnection();
end
data = fetch(j.Connection, "SELECT * FROM mycatalog.myschema.mytable LIMIT 10");
close(j);

JDBC: Write-Optimized Connection

Simba driver write performance improves with native query mode (enabled by default).

j = databricks.JDBCConnection(catalog="main", schema="telemetry");
sqlwrite(j.Connection, "measurements", data);
close(j);

Connection Cleanup

Use onCleanup to guarantee closure even when operations fail.

j = databricks.JDBCConnection(catalog="main", schema="analytics");
cleanup = onCleanup(@() close(j));
data = fetch(j.Connection, "SELECT * FROM large_table WHERE id > 1000");

For Spark stale sessions:

clear spark
spark = getDatabricksSession(forceNewSession=true);

Conventions

• Always filter DataFrames server-side before calling table(DF) — pulling millions of unfiltered rows wastes bandwidth and memory
• Use three-level names for Unity Catalog tables: "catalog.schema.table"
• Use getDatabricksSession() for Spark — never construct sessions manually or mimic PySpark builder patterns
• Use databricks.JDBCConnection or StandaloneJDBCConnection for JDBC — never manually construct JDBC URLs with database()
• Always pass authMethod explicitly — the default chain may trigger unexpected browser prompts
• Never hardcode tokens or secrets in MATLAB code — use .databrickscfg or environment variables
• Always call close(j) when done with JDBC connections
• Use forceNewSession=true for stale Spark sessions, not clear all
• For JDBC driver selection details, see references/driver-selection.md — consult when choosing between Simba and OSS drivers or configuring Java

Common Mistakes

Mistake	Why It's Wrong	Correct Approach
Manually building JDBC URLs with database()	Fragile, error-prone, exposes secrets, misses automatic URL construction, driver classpath management, unified auth chain, and write optimization	Use databricks.JDBCConnection() or StandaloneJDBCConnection()
Inventing PySpark builder: databricks.spark.Session.builder().remote()	Does not exist in MATLAB	Use getDatabricksSession()
Calling table(DF) on unfiltered large tables	Transfers entire dataset locally	Apply .filter() / .select() / .limit() first
Passing scalars to withColumn: DF.withColumn("x", 2)	Second argument must be a Column object	Use DF.withColumn("x", lit(2))
Writing without .mode(): DF.write.save(path)	Fails if target exists	Use .mode("overwrite") or .mode("append")
Using driver auth on-cluster	Browser OAuth fails in browser-based MATLAB	Use useDriverAuth=false
Hardcoding tokens in source code	Security risk; tokens expire	Store in .databrickscfg or environment variables
Calling getDatabricksSession() without authMethod	Default chain may trigger unwanted browser prompt	Pass authMethod="PAT" or other method explicitly
Mismatched Python version for Spark	Session creation fails	Match local Python to runtime (e.g., 3.12 for runtime 17.3)
Using OSS driver with Java 8	OSS requires Java 11+	Set Java 11+ via jenv first (requires MATLAB restart)
Passing DataFrame args to chained methods: DF.unionAll(DF2)	MATLAB Spark wrapper doesn't support DataFrame method arguments	Use spark.sql() with SQL UNION ALL instead

---

Copyright 2026 The MathWorks, Inc.

---