| name | airflow-spark-lakehouse |
| description | Operate Anton's Airflow-owned Spark lakehouse workflow. Use for Workflow Run or Spark Attempt diagnosis, Flight Recorder runs, exact-window retries, shadow-gate evidence, writer cutover, retention, or cleanup. |
Airflow Spark Lakehouse
Goal: Operate one Airflow-owned lakehouse workflow without losing identity,
evidence, or single-writer safety.
Success means:
- One Workflow Run and Spark Attempt are traced by exact identity.
- Retry, Lease, Trino, Loki, and History Server evidence agrees.
- Any writer transfer preserves one authoritative writer.
Stop when: the selected workflow reaches its evidence gate, or another
authority boundary requires operator approval.
Authority
- ADR 0039 owns current platform retention and service-level boundaries.
- ADR 0033 records the Airflow and Apache Spark Operator architecture.
- ADR 0036 records single-writer ownership and accepted metadata loss.
- Plan 0023 records the completed rollout and writer cutover.
.scratch/airflow-spark-lakehouse/spec.md owns implementation acceptance.images/airflow-runtime/src/anton_airflow/ owns workflow behavior.scripts/lib/airflow_shadow_gate.py owns retained gate validation.scripts/lib/flight_recorder_evidence.py owns live Flight Recorder evidence policy.scripts/airflow-lakehouse.py owns the evidence command boundary.
Use seaweedfs-iceberg-data-access for table-level Spark and Trino work.
Use seaweedfs-iceberg-lakehouse for storage, catalog, and Harbor work.
Use query-kubernetes-logs for detailed Loki queries.
Read the matching branch
- Read workflow-state-and-retry.md
for identity, watch, Lease, retry, cancellation, or source-window work.
- Read shadow-gate-evidence.md for
evidence collection, validation, candidate changes, or gate failures.
- Read flight-recorder-acceptance.md
for a manual Flight Recorder run, replay, dependent image release, or
acceptance check.
- Read live-recovery-scenarios.md for
scheduler, triggerer, retry, cancellation, Lease, or pre-commit tests.
- Read cutover-and-cleanup.md for writer
transfer, observation, legacy removal, or experiment cleanup.
Safety boundaries
- Start with read-only repository and cluster checks.
- Treat
exec, port-forward, Workflow Run creation, and reconcile as live mutations.
- Obtain explicit approval before each live mutation.
- Keep credential values out of commands, output, evidence, and files.
- Keep repository edits, credential work, Flux actions, and data deletion separate.
- Record the exact target, namespace, revision, time window, and cleanup owner.
- Fail closed when application state, Lease ownership, or commit state is ambiguous.
Read-only pulse
mise exec -- kubectl config current-context
mise exec -- flux get ks -A | rg 'airflow|spark-operator|spark-history|trino'
mise exec -- flux get hr -A | rg 'airflow|spark-operator|trino'
mise exec -- kubectl -n airflow get pods
mise exec -- kubectl -n lakehouse get sparkapplications,pods,leases
Compare expected state with ADR 0039, deployed source, and the current task.
Use Plan 0023 only for migration history.
Use the exact-attempt collector for retained and live evidence:
mise exec -- task airflow:attempt-evidence \
RUN_ID=<exact-run-id> TARGET=authoritative
Diagnose one workflow
- Record the DAG ID, run ID, task ID, map index, and try number.
- Record the target, source window, candidate revision, and image digests.
- Resolve the exact Spark Attempt and writer Lease.
- Trace Airflow, SparkApplication, driver, executor, Trino, Loki, and history evidence.
- Stop at the earliest failing layer.
- Select the matching reference before proposing a retry or mutation.
Use generation, identity, revision, and verified dependencies to connect events.
Timestamp proximity alone does not prove causality.
Repository validation
mise exec -- python3 -m unittest scripts.tests.test_airflow_shadow_gate
mise exec -- python3 -m unittest scripts.tests.test_airflow_lakehouse_operations
mise exec -- task airflow:shadow-gate \
LEDGER=.scratch/airflow-spark-lakehouse/evidence/shadow-gate-20260813-rotated/ledger.json
mise exec -- task contracts:validate
Repository checks do not authorize a Workflow Run, reconcile, credential change,
writer cutover, or data deletion.
Report format
Status: passed | failed | blocked
Workflow Run: <DAG, run, task, try, target>
Spark Attempt: <name, state, Lease holder>
Evidence: <Trino, Loki, History Server, candidate, credential epoch>
Cause: <earliest supported failing layer>
Changes: <repository paths only>
Approval needed: <exact live or external action, if any>
Residual risk: <unverified recurrence or retention window>
