write-audit-publish

name	write-audit-publish
description	Engine-agnostic advisor for the Write-Audit-Publish (WAP) data-quality pattern — stage an isolated write, audit it, and atomically publish only clean data so consumers never see bad data. Use when designing or reviewing a pipeline that needs a quality gate before exposing results, or when someone mentions WAP, write-audit-publish, staging-then-publish, audit gate, atomic swap/publish, blue-green data, Iceberg WAP branch/cherry-pick, "don't publish bad data", "validate before exposing", or quality SLAs on a published dataset. Advises on design and mechanism choice; does not generate production pipeline code. Not for generic dbt test writing unrelated to a publish gate, dimensional modeling, or ingestion/CDC design.
allowed-tools	Read Grep Glob
license	CC-BY-4.0
status	unstable
metadata	{"author":"Leandro Kellermann de Oliveira <[e-mail]>","version":"0.1.0","model":"sonnet"}

Write-Audit-Publish (WAP) Advisor

WAP keeps bad data from ever becoming visible to consumers: write the new data to an isolated (staging/hidden/branch) location, run audits against it, and only atomically publish if every blocking check passes. The atomic publish is the crux — it flips a pointer with no partial-visibility window. WAP was popularized by Netflix (2017) and is the canonical safety gate for any published dataset with quality SLAs.

This skill advises on design decisions and mechanism choice. It does not generate production pipeline code. Walk the engineer through the fit gate, then the three phases, then failure handling. Recommend, explain trade-offs, and point to the right atomic-publish primitive for their engine.

Use this workflow

Apply the steps in order. Steps 1 and 4 are where engineers most often get WAP wrong, so never skip them.

Step 1 — Confirm WAP actually fits (the gate)

WAP earns its complexity only when two preconditions hold. Check both before recommending it:

1. The engine can publish atomically and cheaply. WAP depends on flipping a pointer (branch fast-forward, metadata table swap, view repoint, partition swap) with no window where consumers see partial data. If the only "publish" available is a row-by-row copy or a TRUNCATE+INSERT, WAP degrades into a risky two-phase write — recommend a different guard (e.g. a circuit breaker on the write) instead. See the engine references under references/engines/ to check what primitive the engine offers.
2. The write is idempotent. Re-running a failed or retried job must produce the same result (partition-overwrite or deterministic merge keys). WAP on a non-idempotent write just stages corruption. If idempotency isn't in place, fix that first.

Also confirm WAP is worth it: the dataset has real consumers and a quality SLA. For a private scratch table no one queries, the audit gate is overhead. Say so.

If the gate fails, say which precondition is missing and what to do instead — don't force WAP.

Step 2 — Design the Write phase (isolation)

Goal: produce the new data in a location consumers cannot see yet, using the cheapest isolation the engine offers. In priority order:

• Branch / snapshot isolation (Iceberg WAP branch, lakeFS/Nessie branch, Delta clone) — best: zero-copy, isolated, atomic to merge.
• Staging table/schema swapped at publish (Snowflake SWAP WITH, BigQuery/Redshift staging dataset, blue-green via a view).
• Staging partition/prefix swapped into place (Hive partition SET LOCATION, object-store manifest/pointer).

Pin the isolation to the same primitive you'll publish with in Step 4 — they're two ends of one mechanism. Make the write idempotent (overwrite the staging target, don't append blindly).

Step 3 — Design the Audit phase (the checks)

Audits run against the isolated staging data, never against production. Design a check suite and classify each check as blocking (failure halts publish) or warning (logged, publish proceeds). Cover the families:

• Volume / row-count (absolute floor, delta vs. previous run)
• Reconciliation / control totals vs. source (catches silent loss that shape checks miss)
• Completeness / null checks on required columns
• Uniqueness / primary-key / duplicate checks
• Schema conformance (expected columns, types, no unexpected drift)
• Referential integrity (FKs resolve to dimensions)
• Range / domain / accepted-values
• Freshness / recency within SLA
• Business-rule invariants (e.g. amount >= 0, segment totals reconcile)

For the full catalog, blocking-vs-warning guidance, and tool pointers (dbt tests, dbt-expectations, Great Expectations, Soda, Deequ/PyDeequ), read references/audit-checks.md.

Step 4 — Design the Publish phase (atomic swap)

This is the part that makes it WAP and not "stage and pray." Pick the engine's atomic primitive so consumers transition from old → new in a single metadata operation with no partial-visibility window:

Engine / platform	Atomic publish primitive	Reference
Apache Iceberg	Write to a WAP branch, audit the snapshot, fast-forward / cherry-pick to main	references/engines/iceberg.md
Delta Lake / Spark	Staging table or shallow clone, then atomic metadata swap / CREATE OR REPLACE	references/engines/delta.md
Snowflake (dbt)	ALTER TABLE … SWAP WITH (atomic name swap) or zero-copy clone	references/engines/snowflake.md
BigQuery (dbt)	CREATE OR REPLACE TABLE from staging, or repoint a view	references/engines/bigquery.md
Redshift (dbt)	Staging table swap inside a transaction (DDL is transactional)	references/engines/redshift.md
Hive / partitioned lake	ALTER TABLE … PARTITION … SET LOCATION (directory swap)	references/engines/hive.md
Object storage (S3/GCS)	Atomically update a manifest / pointer / symlink object	references/engines/object-storage.md
lakeFS / Nessie	Merge the audited branch into main (git-like atomic merge)	references/engines/lakefs-nessie.md
Any SQL warehouse	Blue-green: repoint a consumer view between tbl_a / tbl_b	references/engines/view-indirection.md

Read the matching references/engines/<engine>.md for exact commands, gotchas, and atomicity caveats before recommending an implementation.

If no atomic primitive exists — the engine offers only TRUNCATE+INSERT, append, or row-by-row upsert with no transaction or pointer-swap — true WAP is not achievable; any "publish" leaves a partial-visibility window. Recommend instead: a circuit breaker on the write (audit staging, abort the write entirely on failure, accepting there's no atomic flip); introducing a view layer (references/engines/view-indirection.md) if the engine has views, often the cheapest way to gain an atomic primitive; or migrating the published surface to a table format (Iceberg/Delta) or branch tool (lakeFS/Nessie) if the quality SLA justifies it. Be explicit that without an atomic primitive the guarantee is weaker than real WAP.

Step 5 — Design failure handling

When a blocking audit fails, publish must not happen. Decide:

• Circuit breaker: below the quality threshold, halt and alert rather than publish degraded data. WAP and the circuit breaker pair naturally — WAP is where the breaker trips.
• Quarantine, don't discard: keep the failed staging data (branch/table/partition) for inspection and replay instead of deleting it. Treat it like a dead-letter for the whole batch.
• Alerting & ownership: who is paged, with what context (which check, observed vs. expected).
• Retention / cleanup: when stale staging branches/tables/partitions get garbage-collected so they don't accumulate cost.
• Idempotent retry: because the write is idempotent (Step 1), a retry safely re-stages and re-audits from scratch.

Anti-patterns to flag

• Auditing production instead of staging. If checks run after publish, consumers have already seen bad data. Audit the isolated copy.
• Non-atomic "publish." A TRUNCATE+INSERT, a multi-statement copy, or per-row upsert leaves a partial-visibility window — that's not WAP. Insist on a real atomic primitive (Step 4).
• WAP on a non-idempotent write. Retries duplicate or corrupt the staging data; the gate gives false confidence.
• All checks blocking (or none). Everything-blocking makes the pipeline brittle and noisy; nothing-blocking makes WAP decorative. Classify deliberately (Step 3).
• Deleting failed batches. Destroys the evidence needed to debug and replay. Quarantine instead.
• Reusing the same physical target for stage and serve. Defeats isolation. Stage must be invisible to consumers until publish.

Examples

Example 1 — "We're on Iceberg and want to stop publishing bad partitions"

1. Gate (Step 1): Iceberg has native branch WAP and partition-overwrite idempotency — both preconditions hold. Proceed.
2. Write: enable write.wap.enabled, write the run to an audit branch.
3. Audit: row-count floor + null checks on keys + reconciliation vs. source, all blocking; distribution shift as warning. (See references/audit-checks.md.)
4. Publish: cherry-pick / fast-forward the audited snapshot to main. (See references/engines/iceberg.md.)
5. Failure: on a blocking failure, leave the audit branch in place, page the owner, GC the branch after N days.

Example 2 — "Plain Postgres warehouse, no branching — can we still do WAP?"

1. Gate: Postgres has no branch primitive, but DDL is transactional and views give a clean atomic repoint — atomic publish is available. Idempotency: confirm the load overwrites its target deterministically. If yes, proceed.
2. Write: build into orders_staging.
3. Audit: control totals vs. source + PK uniqueness + freshness, blocking.
4. Publish: blue-green — CREATE OR REPLACE VIEW orders AS SELECT * FROM orders_staging (or swap table names in a transaction). Consumers query the view and never see the build. (See references/engines/view-indirection.md.)
5. Failure: keep orders_staging, alert, don't repoint the view.

Example 3 — "Should we add WAP to this scratch table?"

Push back. Run the gate (Step 1): if the table has no SLA-bound consumers, the audit gate is pure overhead. Recommend lightweight inline assertions or a circuit breaker on the write instead, and reserve WAP for the published datasets downstream.

Reference routing

• Choosing or implementing the atomic swap → the engine table in Step 4, then the matching references/engines/<engine>.md (iceberg, delta, snowflake, bigquery, redshift, hive, object-storage, lakefs-nessie, view-indirection)
• Designing the audit suite and classifying checks → references/audit-checks.md