| name | azure-databricks |
| description | Databricks on Azure: cluster lifecycle, Delta Lake operations, MLflow tracking, and Unity Catalog patterns for this repo. |
| user-invocable | false |
| allowed-tools | ["Bash(databricks *)","WebFetch(docs.databricks.com)"] |
Azure Databricks Domain Guide
This repo uses Azure Databricks. Apply these patterns.
Cluster Lifecycle
- Never create interactive clusters when job clusters suffice — job clusters are cheaper and auto-terminate.
- Pin cluster runtime versions explicitly; do not rely on
latest.
- Check cluster state before submitting jobs:
databricks clusters get --cluster-id <id>.
- Auto-termination: always set
autotermination_minutes on interactive clusters.
Delta Lake
- Always read with
spark.read.format("delta") — not parquet — for managed Delta tables.
- Schema evolution: use
.option("mergeSchema", "true") only when schema drift is intentional.
- Vacuuming:
VACUUM tablename RETAIN 168 HOURS — never vacuum below the 7-day default without explicit approval.
- Z-ORDER by query columns, not partition columns.
MLflow
- Log every experiment run:
mlflow.set_experiment("name") before mlflow.start_run().
- Log parameters, metrics, and artifacts explicitly — autolog misses edge cases.
- Registered models go in Unity Catalog:
models:/CatalogName.SchemaName.ModelName/version.
Unity Catalog
- Use three-part names:
catalog.schema.table.
- Never use
hive_metastore for new tables — that's the legacy path.
- Check grants before assuming read/write access:
SHOW GRANTS ON TABLE catalog.schema.table.
Safety
- Treat production clusters as read-first. Never write to production Delta tables without explicit approval.
- Databricks secrets must be stored in secret scopes, not in notebook cells or job configs.
- Use service principals for automation — not personal access tokens committed to source.
