Guard production Databricks data pipelines — Delta Lake, Liquid Clustering,
Structured Streaming, Auto Loader, and DLT — against the twelve foot-guns that
fire at scale: OPTIMIZE/auto-compaction conflicts, Liquid-Clustering merge
conflicts, VACUUM breaking a streaming checkpoint, RocksDB OOM, Auto Loader
schema-evolution stops, and DLT refresh data loss. Includes a PreToolUse hook
that blocks DROP/CREATE-OR-REPLACE/VACUUM on a table with active streaming
consumers. Use when a Delta MERGE/OPTIMIZE fails with a concurrency exception, a
stream breaks after VACUUM or a table replace, an Auto Loader stream stops on a
new column, a DLT refresh drops data, or before running a destructive op on a
streamed-from table. Trigger with "ConcurrentAppendException",
"ConcurrentDeleteDeleteException", "DELTA_FILE_NOT_FOUND", "streaming checkpoint
broke", "vacuum broke my stream", "autoloader UnknownFieldException", "dlt full
refresh".
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Guard production Databricks data pipelines — Delta Lake, Liquid Clustering,
Structured Streaming, Auto Loader, and DLT — against the twelve foot-guns that
fire at scale: OPTIMIZE/auto-compaction conflicts, Liquid-Clustering merge
conflicts, VACUUM breaking a streaming checkpoint, RocksDB OOM, Auto Loader
schema-evolution stops, and DLT refresh data loss. Includes a PreToolUse hook
that blocks DROP/CREATE-OR-REPLACE/VACUUM on a table with active streaming
consumers. Use when a Delta MERGE/OPTIMIZE fails with a concurrency exception, a
stream breaks after VACUUM or a table replace, an Auto Loader stream stops on a
new column, a DLT refresh drops data, or before running a destructive op on a
streamed-from table. Trigger with "ConcurrentAppendException",
"ConcurrentDeleteDeleteException", "DELTA_FILE_NOT_FOUND", "streaming checkpoint
broke", "vacuum broke my stream", "autoloader UnknownFieldException", "dlt full
refresh".
The data-ops spine of the pack. Delta Lake, Liquid Clustering, Structured
Streaming, and DLT each ship a different set of foot-guns that fire most visibly
when production data flows through them at scale — and most of them are documented
platform decisions that surprise engineers, not bugs. This skill's job is
friction at trigger time (a hook that blocks the genuinely-irreversible op) plus
deterministic recovery when something already broke.
Overview
Twelve foot-guns, grouped by the surface that triggers them. Eleven are owned
outright (D01–D10, D12); the twelfth — D11, DLT rebuild cost — is shared with
databricks-cost-leak-hunter: this skill checks the rebuild cost as part of
pre-refresh safety, that skill owns ongoing cost optimization.
Delta write conflicts. D01 ConcurrentDeleteDeleteException — a manual
OPTIMIZE colliding with auto-compaction, which is silently enabled on any table
touched by MERGE/UPDATE/DELETE. D02 ConcurrentAppendException after moving
to Liquid Clustering — LC keeps file-set-level writer conflicts; a fan-out MERGE
breaks unless its predicate is narrowed to the clustering keys.
Streaming + checkpoint. D03 DELTA_FILE_NOT_FOUND_DETAILED — VACUUM deletes
files the checkpoint pins to. D04 silent checkpoint corruption / reset to batch 0.
D05 RocksDB state-store off-heap OOM (the heap looks fine while off-heap state pins
multi-GB). D12 DIFFERENT_DELTA_TABLE_READ_BY_STREAMING_SOURCE — CREATE OR REPLACE mints a new UUID and kills every active consumer.
Migration + evolution. D06 Liquid-Clustering migration's hidden full-rewrite
cost + downstream partition-predicate breakage. D07 time travel breaking silently
when VACUUM crosses the retention boundary. D10 Auto Loader
UnknownFieldException stopping the stream on every new column.
DLT. D08 the @dlt.table thread race (out-of-order registration). D09 full
refresh silently dropping data from a non-replayable source. D11 the rebuild cost
multiplier (checked before a full refresh; ongoing DLT cost is
databricks-cost-leak-hunter's job).
The hook (AP02/AP06 — this pack's only blocking hook). A PreToolUse hook
(hooks/streaming-guard-hook.py) intercepts a Bash command that runs DROP TABLE,
CREATE OR REPLACE TABLE, or against a table and — only when it confirms
via that an active stream reads that table —
with a message naming the consumers and the pain. It is precise by
design: it matches only real SQL-execution surfaces (never a mentioning
"drop table"), and it — if it cannot verify consumers, it allows
rather than false-block. Blocking is reserved for the genuinely irreversible.
VACUUM
system.streaming.query_progress
blocks it
git commit
fails open
Deterministic work lives in scripts/; deep knowledge in references/; the
Liquid-Clustering predicate rewrite in the merge-rewriter subagent. Two data
planes: the databricks-workspace-mcp control plane (cluster/pipeline events) and
the CLI Statement Execution API for system.* reads. Either absent → advisory mode
on pasted input.
Prerequisites
databricks-workspace-mcp registered — for clusters_events (RocksDB OOM
correlation) and pipelines_get (DLT event log). Absent → advisory mode.
Databricks CLI authenticated + jq, and DATABRICKS_WAREHOUSE_ID set —
for the system.streaming.query_progress reads the hook and recovery flows use.
The hook fails open (allows) if these are absent, so it never false-blocks.
The hook is a plugin-level PreToolUse hook — it runs on Bash commands once
the pack is installed. It is silent on everything except a confirmed-unsafe
destructive op.
Instructions
Pick the flow by symptom. Always name the exact, searchable Databricks error
string — ConcurrentAppendException, ConcurrentDeleteDeleteException,
DELTA_FILE_NOT_FOUND_DETAILED, DIFFERENT_DELTA_TABLE_READ_BY_STREAMING_SOURCE,
UnknownFieldException — even when the user paraphrases it or gives a short form;
the full code is what an operator greps logs and docs for.
Step 1: Before a destructive op (the hook does this automatically)
Running DROP TABLE / CREATE OR REPLACE TABLE / VACUUM on a table? The hook
checks for active streaming consumers first and blocks if any exist. To check
manually, query system.streaming.query_progress for a stream whose
source_description names the table. If consumers exist: do NOT CREATE OR REPLACE (use ALTER/in-place — D12) and do NOT VACUUM below the consumers'
checkpoint lag (D03/D07). See
${CLAUDE_SKILL_DIR}/references/checkpoint-recovery.md.
Step 2: A Delta write conflict (D01, D02)
ConcurrentDeleteDeleteException (D01) — before a manual OPTIMIZE, probe
the table for auto-compaction:
ConcurrentAppendException on a Liquid-Clustering table (D02) — hand the
failing MERGE to the merge-rewriter subagent; it fetches the target's
clustering keys via DESCRIBE DETAIL and narrows the ON predicate so writers
touch disjoint file sets.
Step 3: A broken streaming source (D03, D04, D12)
Map the symptom to the failure class and its exact error code, then get the
recovery tier from the decision tree:
It echoes the canonical error code and recommends SAFE_RESTART /
REPROCESS_FROM_OFFSET / RESTORE_FROM_TIME_TRAVEL / FULL_RESET_BACKFILL with the
data-loss tradeoff stated up front. Name that full code in your answer — not just
the short class. The full
three-tier reasoning is in
${CLAUDE_SKILL_DIR}/references/checkpoint-recovery.md.
Step 4: RocksDB state-store OOM (D05)
A driver/executor OOM while the JVM heap looks healthy points at off-heap RocksDB
state. Correlate the OOM to state size with clusters_events, then bound the
memory and enable changelog checkpointing per
${CLAUDE_SKILL_DIR}/references/rocksdb-state-store-tuning.md.
Step 5: Auto Loader schema evolution (D10)
A stream stopping with UnknownFieldException on a new column is the default
addNewColumns mode. Choose the mode deliberately (evolve-and-restart vs
rescue's silent widening) and pin types with schemaHints per
${CLAUDE_SKILL_DIR}/references/autoloader-schema-evolution.md.
Step 6: DLT rebuild safety (D08, D09, D11)
Before a DLT full refresh, confirm every source is replayable (a Kafka topic past
retention or a truncate-and-load source loses data on refresh — D09) and that
@dlt.table registration is deterministic (the thread race — D08). Read the DLT
event log with pipelines_get; the checklist is in
${CLAUDE_SKILL_DIR}/references/dlt-rebuild-safety.md.
Output
A hook decision — a destructive op on a streamed-from table is blocked with
the active consumers named and the pain (D12 / D03-D07) explained; everything
else passes silently.
A pre-OPTIMIZE verdict — SAFE or COLLISION RISK (auto-compaction on) with the
disable-or-serialize fix.
A rewritten MERGE — the LC clustering-key-scoped predicate (from
merge-rewriter) that stops ConcurrentAppendException.
A recovery recommendation — the recovery tier + steps + the data-loss risk,
for the specific failure class.
A tuning / mode / refresh-safety recommendation — RocksDB bounds, Auto Loader
mode, or the DLT full-refresh checklist, from the matching reference.
Error Handling
Error
Cause
Solution
ConcurrentDeleteDeleteException
Manual OPTIMIZE races auto-compaction (D01)
Run pre-optimize-check.sh; don't manually OPTIMIZE an auto-compacted table, or disable auto-compact first.
ConcurrentAppendException on an LC table
MERGE predicate not scoped to clustering keys (D02)
Route the MERGE to merge-rewriter; narrow the ON predicate to the clustering keys.
DELTA_FILE_NOT_FOUND_DETAILED
VACUUM deleted checkpoint-pinned files (D03)
recover-streaming-source.py --failure file-not-found; restore via time travel if in retention, else reprocess/reset.
DIFFERENT_DELTA_TABLE_READ_BY_STREAMING_SOURCE
CREATE OR REPLACE minted a new UUID (D12)
The old checkpoint is dead — new checkpoint + backfill; the hook prevents this going forward.
Driver OOM, heap looks fine
Off-heap RocksDB state (D05)
Bound state-store memory + changelog checkpointing; size state with a watermark.
UnknownFieldException, stream stopped
Auto Loader addNewColumns default (D10)
Restart to evolve (idempotent sink), or choose rescue/schemaHints deliberately.
Hook allowed a destructive op with a warning
Could not verify consumers (no CLI/warehouse)
Advisory — the hook fails open; verify system.streaming.query_progress manually before running it.
Examples
Example 1: "About to CREATE OR REPLACE a table other jobs stream from."
The PreToolUse hook fires, confirms 2 active consumers via
system.streaming.query_progress, and blocks with: "CREATE OR REPLACE mints a
new UUID → both consumers die with DIFFERENT_DELTA_TABLE_READ_BY_STREAMING_SOURCE;
use ALTER / in-place."
Example 2: "My MERGE into a Liquid-Clustering table fails with ConcurrentAppendException."
The merge-rewriter subagent reads the target's clustering keys via DESCRIBE DETAIL and rewrites the ON predicate to include them, so concurrent writers
touch disjoint file sets — the exception stops without serializing the jobs.
Example 3: "My stream died with DELTA_FILE_NOT_FOUND after a VACUUM."
recover-streaming-source.py --failure file-not-found --time-travel yes →
RESTORE_FROM_TIME_TRAVEL (no data loss): restore the source to a pre-VACUUM version,
restart on the existing checkpoint, then align VACUUM retention with the checkpoint lag.
Example 4: "Before I OPTIMIZE this table."
pre-optimize-check.sh --table main.sales.orders reports COLLISION RISK because
delta.autoOptimize.autoCompact is on — so the skill recommends letting
auto-compaction do it, or disabling it for the maintenance window first.
