Connect MATLAB to Databricks via Spark (Databricks Connect) or JDBC (Database Toolbox). Use when setting up the MATLAB Interface for Databricks, configuring authentication (OauthU2M, OauthM2M, PAT), creating Spark sessions with getDatabricksSession(), reading Unity Catalog tables with server-side filtering, creating JDBC connections with databricks.JDBCConnection or StandaloneJDBCConnection, connecting to SQL Warehouses, selecting JDBC drivers (Simba/OSS), or writing data back to Databricks. Triggers on: Databricks Connect, Spark from MATLAB, getDatabricksSession, .databrickscfg, databricks.JDBCConnection, StandaloneJDBCConnection, SQLWarehouse, Databricks JDBC, Databricks cluster, large table server-side filtering.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
Instruções da origem · Visualização somente leitura
name
matlab-connect-databricks
description
Connect MATLAB to Databricks via Spark (Databricks Connect) or JDBC (Database Toolbox). Use when setting up the MATLAB Interface for Databricks, configuring authentication (OauthU2M, OauthM2M, PAT), creating Spark sessions with getDatabricksSession(), reading Unity Catalog tables with server-side filtering, creating JDBC connections with databricks.JDBCConnection or StandaloneJDBCConnection, connecting to SQL Warehouses, selecting JDBC drivers (Simba/OSS), or writing data back to Databricks. Triggers on: Databricks Connect, Spark from MATLAB, getDatabricksSession, .databrickscfg, databricks.JDBCConnection, StandaloneJDBCConnection, SQLWarehouse, Databricks JDBC, Databricks cluster, large table server-side filtering.
Connect MATLAB to Databricks via Spark (Databricks Connect) or JDBC (Database Toolbox). This skill covers first-time setup, authentication, path selection, session/connection creation, and data operations through both paths.
When to Use
First-time setup of the MATLAB Interface for Databricks package
Configuring .databrickscfg and authentication (OauthU2M, OauthM2M, PAT)
Choosing between Spark and JDBC for a Databricks workflow
Reading large tables via Spark with server-side filtering (getDatabricksSession)
Creating JDBC connections to clusters or SQL Warehouses (, )
databricks.JDBCConnection
SQLWarehouse.connect()
Standalone JDBC connectivity without the full package (StandaloneJDBCConnection)
Writing data back to Databricks (via Spark DataFrames or JDBC sqlwrite)
Selecting and configuring JDBC drivers (Simba vs. OSS)
When NOT to Use
Generic Database Toolbox operations after connection is established — use matlab-use-database
Deploying compiled MATLAB code to Databricks clusters (Job workflow)
DuckDB — use matlab-use-duckdb
Databricks notebooks, Databricks CLI, or databricks-sdk Python workflows
PySpark without MATLAB context (pure Python Spark usage)
Decision Framework: Spark vs. JDBC
Scenario
Path
Why
Large table (millions of rows), need server-side filtering before pulling locally
Spark
DataFrame operations run on cluster; only filtered results transfer
SQL queries on small-to-medium datasets
JDBC
Direct SQL via Database Toolbox; simpler setup
Need DataFrame transformations (withColumn, select, filter chains)
Spark
Native DataFrame API; operations stay on cluster
Writing large data with performance optimization
JDBC
Simba driver's UseNativeQuery optimizes sqlwrite
No MATLAB Interface for Databricks package installed
JDBC
StandaloneJDBCConnection works with just Database Toolbox + driver jar
Need to use Database Explorer app
JDBC
saveSource() + copyToken() integration
Reading files from Unity Catalog Volumes (CSV, Parquet, JSON)
Spark
spark.read.format().load() with Volumes paths
Interactive exploration with sqlread/fetch
JDBC
Standard Database Toolbox workflow on j.Connection
Default: Use Spark when the user mentions large data, server-side filtering, or DataFrames. Use JDBC when the user mentions SQL queries, Database Toolbox, or small/medium datasets. If unclear, ask the user about their data size and preferred workflow.
Setup — Run setup from the package's Software/MATLAB directory. It configures settings, .databrickscfg, and installs the Databricks Connect library via pip into a venv
Startup — Run startup to add package paths (required once per MATLAB session)
Choose path — Use the Decision Framework above to select Spark or JDBC
Connect — Create a session (getDatabricksSession) or connection (databricks.JDBCConnection)
Verify — Spark: table(spark.range(1)). JDBC: check j.Connection.Message is empty
Operate — Read, filter, write data using the appropriate path's API
Close — JDBC: close(j). Spark: sessions are managed automatically
Key Functions
Spark Path
Function
Purpose
getDatabricksSession()
Creates a Spark session (classic compute)
getDatabricksSession(serverless=true)
Creates a serverless session (no cluster, 10-min timeout)
spark.read().table("catalog.schema.table")
Reads a Unity Catalog table as a DataFrame
spark.read.format(fmt).load(path)
Reads files from Volumes (csv, parquet, json)
DF.filter(expr)
Server-side row filtering
DF.select(cols)
Server-side column selection
DF.limit(n)
Server-side row limiting
DF.withColumn(name, col)
Adds/transforms a column (requires Column objects)
table(DF)
Converts DataFrame to MATLAB table (pulls data locally)
matlab.sparkutils.table2dataset(T, spark)
Converts MATLAB table back to Spark DataFrame
DF.write.mode(m).format(f).saveAsTable(name)
Writes DataFrame to Unity Catalog table
matlab.pyspark.sql.functions.col(name)
Creates a Column reference
matlab.pyspark.sql.functions.lit(value)
Creates a literal Column constant
JDBC Path
Function
Purpose
databricks.JDBCConnection()
Creates a JDBC connection (full package)
StandaloneJDBCConnection()
Creates a JDBC connection (no package dependencies)
databricks.SQLWarehouse.connect()
Connects to a SQL Warehouse by ID
j.Connection
The database.jdbc.connection object for Database Toolbox functions
j.testConnection()
Verifies connection is working
j.saveSource()
Saves connection as a Database Toolbox data source
close(j)
Closes connection and releases resources
Patterns
First-Time Setup
Run setup from the package's Software/MATLAB directory. It is interactive — follow prompts for host URL, auth method, cluster ID, and Databricks Connect library installation.
The Databricks Connect library is downloaded via pip into Software/MATLAB/Connect/<version>/venv/. Requires Python 3.10-3.12. If the download fails repeatedly, ask the user to check with their IT team — do not retry with modified arguments.
Warning: If Python is already loaded InProcess, terminate(pyenv) fails. A full MATLAB restart is required. Do not attempt to switch pyenv mid-session after Python has been used.
For authentication configuration details (.databrickscfg format, profiles, environment variables, token caching), see references/authentication.md — consult when configuring auth methods or troubleshooting credential issues.
Spark: Read and Filter a Table
Data stays on the cluster until explicitly collected. Filter server-side first, then collect.
spark = getDatabricksSession(authMethod="PAT");
DF = spark.read().table("catalog.schema.sensor_readings");
filtered = DF.filter("temperature > 100 AND event_date > '2024-01-01'");
T = table(filtered);
Spark: Serverless Session
No cluster needed. Starts instantly with 10-minute inactivity timeout. Requires Python 3.11-3.12 and Databricks Connect >= 15.4.
spark = getDatabricksSession(serverless=true);
DF = spark.read().table("catalog.schema.events");
T = table(DF.limit(50));
Spark: Add Computed Columns
withColumn requires Column objects — not raw scalars. Use col() for references and lit() for constants.
DF = spark.read.format("csv").option("header", "true").load("/Volumes/catalog/schema/volume/data.csv");
T = table(DF.limit(1000));
JDBC: Cluster Connection
j = databricks.JDBCConnection(catalog="mycatalog", schema="myschema");
data = sqlread(j.Connection, "mytable");
close(j);
JDBC: SQL Warehouse Connection
warehouse.connect() returns a database.jdbc.connection directly — pass it to sqlread/fetch without .Connection.
warehouse = databricks.SQLWarehouse;
warehouse.id = "abc123def456";
conn = warehouse.connect();
data = fetch(conn, "SELECT * FROM mycatalog.myschema.mytable LIMIT 10");
close(conn);
JDBC: Standalone (No Package)
When the user does NOT have the MATLAB Interface for Databricks, use StandaloneJDBCConnection. Requires Database Toolbox and the Simba driver jar only. See references/standalone-jdbc.md for setup and JSON template.
j = StandaloneJDBCConnection(schema="myschema", catalog="mycatalog");
data = fetch(j.Connection, "SELECT * FROM mytable LIMIT 10");
close(j);
JDBC: On-Databricks (Browser MATLAB)
The JDBC driver's OAuth flow cannot open a browser when MATLAB runs on a Databricks cluster. Use package-managed auth instead.
if databricks.internal.isOnDatabricks()
j = databricks.JDBCConnection(authMethod="OauthU2M", useDriverAuth=false);
else
j = databricks.JDBCConnection();
end
data = fetch(j.Connection, "SELECT * FROM mycatalog.myschema.mytable LIMIT 10");
close(j);
JDBC: Write-Optimized Connection
Simba driver write performance improves with native query mode (enabled by default).
Use onCleanup to guarantee closure even when operations fail.
j = databricks.JDBCConnection(catalog="main", schema="analytics");
cleanup = onCleanup(@() close(j));
data = fetch(j.Connection, "SELECT * FROM large_table WHERE id > 1000");
