azure-databricks

name	azure-databricks
description	Databricks on Azure: cluster lifecycle, Delta Lake operations, MLflow tracking, and Unity Catalog patterns for this repo.
user-invocable	false
allowed-tools	["Bash(databricks *)","WebFetch(docs.databricks.com)"]

Azure Databricks Domain Guide

This repo uses Azure Databricks. Apply these patterns.

Cluster Lifecycle

• Never create interactive clusters when job clusters suffice — job clusters are cheaper and auto-terminate.
• Pin cluster runtime versions explicitly; do not rely on latest.
• Check cluster state before submitting jobs: databricks clusters get --cluster-id <id>.
• Auto-termination: always set autotermination_minutes on interactive clusters.

Delta Lake

• Always read with spark.read.format("delta") — not parquet — for managed Delta tables.
• Schema evolution: use .option("mergeSchema", "true") only when schema drift is intentional.
• Vacuuming: VACUUM tablename RETAIN 168 HOURS — never vacuum below the 7-day default without explicit approval.
• Z-ORDER by query columns, not partition columns.

MLflow

• Log every experiment run: mlflow.set_experiment("name") before mlflow.start_run().
• Log parameters, metrics, and artifacts explicitly — autolog misses edge cases.
• Registered models go in Unity Catalog: models:/CatalogName.SchemaName.ModelName/version.

Unity Catalog

• Use three-part names: catalog.schema.table.
• Never use hive_metastore for new tables — that's the legacy path.
• Check grants before assuming read/write access: SHOW GRANTS ON TABLE catalog.schema.table.

Safety

• Treat production clusters as read-first. Never write to production Delta tables without explicit approval.
• Databricks secrets must be stored in secret scopes, not in notebook cells or job configs.
• Use service principals for automation — not personal access tokens committed to source.