| name | data-table-lifecycle |
| description | Use when deciding whether a table or artifact should keep existing, planning a table drop or deprecation, auditing a lakehouse for dead/zero-consumer tables, or building maintenance-job coverage across many tables. Covers the consumers-or-deprecate discipline (every table names a reader or gets flagged for removal), the drop-durability trap (a get_or_create-style helper silently resurrecting a "dropped" table on its next write), the metadata-vs-physical split of a drop (catalog drop_table does not delete files), and generating maintenance coverage from the catalog instead of a hand-maintained list. Don't use for choosing a storage format or catalog backend, or schema evolution within a live table — that's data-apache-lakehouse. This skill is about whether a table should exist at all, and how to retire it safely once it shouldn't. Prefer the data hub when the right data skill is unclear or the task spans ingest→store→serve. |
Table lifecycle — consumers-or-deprecate discipline
Tables accumulate. Every lakehouse grows tables nobody's using because deleting one feels riskier
than leaving it — until the unused ones are 25% of disk and nobody remembers why. This skill is the
discipline that keeps that from happening: name a consumer or flag for removal, and retire safely
when a table has none. It's the artifact-lifecycle expression of the data hub's principles, not a
replacement for them.
Verified 2026-07-02 on a 110-table production lakehouse during a stabilization engagement.
When to invoke this skill
- Before adding a new table, view, or derived artifact to a pipeline.
- Auditing a lakehouse to find tables nobody reads.
- Planning to drop, rename, or deprecate an existing table.
- Building or reviewing the list of tables a maintenance job (compaction, expiry) covers.
- A nightly job errors with "table does not exist" and nobody's sure if that's expected.
Principles
Each is paired with a falsifiable test — if you can't pass it, the principle is violated, not
"applied differently."
Every artifact names its consumer
A table exists because something reads it. State the consumer when the table is created, and re-prove
it periodically — not by checking the obvious downstream job, but by an adversarial sweep across
every consumer: sibling applications, not just the pipeline that wrote it; and traffic logs (reverse
proxy / API access logs mapping endpoint → table), not just code that imports the table name. A
110-table lakehouse audit that swept only the pipeline code missed 44 tables (~100GB, 25% of disk)
with zero consumers — they were only found by also grepping sibling apps and mapping nginx traffic
logs to table names.
Test: run the adversarial sweep — grep every consumer and check traffic logs for this table's
endpoints. Does anything read it? If the sweep across all of that finds nothing, deprecate; "the
pipeline that writes it" reading its own output doesn't count as a consumer.
Drops are durable only after writers stop (the resurrection trap)
Dropping a table from the catalog is not the end of the story if anything downstream still writes to
it. A get_or_create_table-style helper — common in lakestore client code, meant to make writes
convenient — will silently recreate a dropped table the next time it runs, because from its point of
view "the table doesn't exist yet" and "I should create it" look identical. A drop is only durable
once every writer has actually stopped calling that path; verified as a live near-miss in production.
Test: after dropping, does anything in the codebase still call get_or_create_table (or
equivalent) against this table's name on any schedule? If yes, the drop isn't durable — it's a table
that will reappear on its own.
A drop is metadata-only; physical cleanup is a separate, owned step
catalog.drop_table (e.g. PyIceberg's SqlCatalog) removes the catalog pointer. It does not delete
the underlying data files — the physical prefix still sits on storage, still costing money, still
readable by anything with a direct path. Physical cleanup is a distinct, owned step, not an assumed
side effect of the drop. The safe sequence:
- Gate/stop writers — confirm no writer path still targets this table (see the resurrection trap above).
- Drop — remove the catalog entry.
- Record the physical prefix — write down exactly what storage path the dropped table occupied, before anyone forgets.
- Physical prefix cleanup — a separate, deliberately-run step that deletes the recorded prefix.
- Vacuum — reclaim the underlying object-store space (see
data-pipeline-operations for tombstone/vacuum economics).
Test: after drop_table returns, does the storage bill go down? If no, the drop was metadata-only
and step 4 hasn't happened yet — that's expected, but it needs to be a tracked, owned task, not a gap
nobody notices.
Maintenance coverage is generated, never hand-listed
A hand-maintained list of "tables this maintenance job covers" rots the moment a table is renamed,
added, or dropped and the list isn't updated in lockstep. Measured failure: 6 of 16 hand-listed table
names had gone stale, causing a nightly maintenance job to silently fail with "table does not exist"
for each — nobody noticed because the job kept "succeeding" on the other 10. Generate the coverage
list from the catalog at run time, minus an explicit exclusion file for tables deliberately skipped
(e.g. still under validation on a branch).
Test: rename or drop a table without touching the maintenance job's config. Does the job's
coverage list update itself on the next run, or does it silently skip/error on the stale name? If the
list needed manual editing to stay correct, it's a hand-list, and it will rot.
The dead-write smell
A build stage whose outputs are majority zero-consumer is a symptom that shows up before the "44
zero-consumer tables" audit finding above — catch it earlier. Measured case: an analytics stage built
4 artifacts per tick; the unbounded full-history read feeding them accounted for ~85% of a 6.69GiB
peak, while most of what it produced had no reader at all. "Every artifact names its consumer" applied
at design time would have caught this years before the audit did.
Test: for a build stage producing multiple outputs, what fraction of them have a named consumer
before the stage ships? If most don't, the stage is doing (and paying for) dead work — cut it at
design time, not at the next lakehouse audit.
References
- Physical cleanup and vacuum economics for the object store underneath the catalog (tombstone
thresholds,
garbageThreshold tuning) → data-pipeline-operations, SeaweedFS coexistence notes.
drop_table semantics, catalog pointer mechanics → data-apache-lakehouse.- Shared cross-cutting principles (idempotency, watermarks, bounded memory) → data hub.
