optimize-databricks-costs

name

optimize-databricks-costs

description

Audit and optimize Databricks account, workspace, SQL warehouse, Apps, Jobs, pipeline, and service-principal costs. Use when investigating a Databricks cost spike, explaining DBU or list-cost drivers, identifying expensive users or workloads, proposing fast savings, assessing the workloads and user impacts of changes, implementing explicitly approved cost controls, or verifying that spend dropped afterward.

Optimize Databricks Costs

Run a read-only cost audit first, convert evidence into an impact-aware proposal, and mutate Databricks only after the user approves the exact affected workloads and impacts.

Operating rules

1. Default to read-only commands and SQL.
2. Do not edit, stop, delete, pause, resize, retag, revoke, or create anything during an audit or proposal.
3. Treat generated reports as private. Never commit hosts, IDs, emails, query text, secrets, or customer object names.
4. Price usage with system.billing.list_prices.pricing.effective_list.default. Call it effective list cost, not invoice cost.
5. Include retractions and restatements when summing billing usage; they correct earlier records.
6. Exclude the current day and an ingestion-lag window from baseline comparisons. Inspect partial recent days separately.
7. Attribute cost to resources and identity_metadata, then prove the underlying workload before describing a user or service principal.
8. Prefer reversible changes. Record the before state and rollback command before every approved mutation.

Phase 1: collect evidence

Confirm the Databricks CLI profile and current identity. List accessible warehouses, Apps, Jobs, pipelines, and clusters without changing them.

Resolve the bundled collector relative to this SKILL.md, then run it from any working directory:

python3 <skill-dir>/scripts/audit.py \
  --warehouse-id <warehouse-used-to-run-system-table-queries> \
  --recent-days 14 \
  --output-dir <private-report-directory>

Use --profile <name> for a non-default Databricks CLI profile. Use --dry-run to inspect the SQL without connecting.

If the script cannot access a system table, retain the successful evidence, report the permission gap, and continue with the accessible inventory. Do not broaden permissions during the audit.

Read audit-methodology.md when interpreting the output. Read system-tables.md when adapting a query to the available Databricks schema.

Phase 2: determine the cost drivers

Establish these facts before recommending a change:

• Billing coverage, latest complete date, unpriced rows, and currency
• Recent versus equal prior-period cost and average daily run rate
• Product-level contribution to the increase
• Resource-level contribution: warehouse, App, Job, pipeline, endpoint, or other identifier
• Identity role: available run_as, run_by, owned_by, or created_by metadata
• Workload behavior: cadence, active hours, task time, cache hits, failures, retries, queueing, and data volume
• Current configuration and current active/paused/stopped state

Investigate the highest-cost or highest-delta product first. A frequent high-count workload is not necessarily the largest account-cost driver.

SQL warehouses

Compare billable active hours with query task time. Flag polling or health checks whose interval is shorter than auto-stop, especially when cache-hit rates are high and task time is tiny. Group by warehouse, client application, driver, identity, and statement fingerprint before inspecting sensitive statement text.

Do not recommend downsizing from aggregate latency alone. Isolate bursty or queue-sensitive clients and review their capacity wait, p95 latency, timeouts, scans, and concurrency.

Databricks Apps

Billing proves that an App incurred compute cost; it does not prove whether humans used the App. Inspect App state, ACLs, App-principal activity, recent authorization events, dependencies, and whether another App actually replaces its features.

Stopping an App preserves its deployment and persistent data but makes its URL unavailable and loses in-memory session state. Users normally restart it manually when needed.

Jobs and pipelines

Separate scheduled production runs from one-time deployments, canaries, retries, failed runs, and cancelled runs. Resolve usage_metadata.job_id and job_run_id to the current Job definition and run timeline. Check downstream lineage before pausing anything with a test- or dev-looking name.

Users and service principals

Resolve UUIDs through SCIM before calling them users. Map a service principal to Jobs, Apps, queries, audit events, and billing metadata. Internal token-login and Unity Catalog audit volume from serverless runtimes is not evidence of interactive human activity.

Phase 3: produce the decision package

Create three concise artifacts, using report-template.md:

1. Cost audit: current situation, root cause, evidence, and quantified opportunity.
2. Optimization proposal: prioritized quick changes with estimated savings and confidence.
3. Impact assessment: exact affected workloads, availability/freshness/latency/correctness impacts, prerequisites, and rollback.

Classify each proposal as:

• Recommended: strong evidence, reversible, low impact
• Conditional: worthwhile after a named owner or technical prerequisite confirms safety
• Not recommended: material unresolved impact or weak savings

Avoid procedural delay for its own sake. For a small company, make the fastest reversible change supported by evidence, smoke-test it immediately, and use billing telemetry as confirmation rather than as a reason to postpone action.

Read optimization-playbook.md for change-specific impact checks.

Phase 4: implement only approved items

Treat approval as scoped. If the user approves the Recommended bundle, do not include Conditional or Not recommended items.

For every mutation:

1. Re-read the live resource and verify its identity.
2. Capture its current configuration.
3. Apply the smallest approved change.
4. Re-read the resource and run a proportional smoke test.
5. Record the timestamp, before/after state, observed impact, and rollback command.

Never delete an App or Job when stop or pause meets the approved objective. Never revoke a human credential as a cost-control shortcut when other workloads may share it.

Phase 5: verify the result

Verify resource state immediately. Query the latest available billing data, but label the current and previous day as partial when applicable. Compare complete post-change days with the earlier run rate without double-counting savings from changes already made.

If cost remains high, repeat attribution from product to resource to workload. Do not assume the original root cause returned.