umd-ci-hatchet-deployment

name

umd-ci-hatchet-deployment

description

UMD GitHub Actions CI reality and Hatchet worker/scheduler deployment pins — run 33164294061 job results, the 14 postgres-job failures (python-multipart/ffmpeg/pg_dump/compose-interpolation, all environment defects with uncommitted working-tree fixes), the docker-e2e compose-up failure, the REAL ghcr.io Hatchet image paths (hatchet-engine/hatchet-lite/hatchet-admin/hatchet-migrate vs the WRONG top-level hatchet-dev/hatchet pin in deploy/compose.yaml:104), SDK 1.38.1 ↔ server v0.105.2 pairing reality, the app.py DurableDAGRunner interim wiring that makes test_api_boundary_e2e skip, the CapabilityReporter connectivity-probe gap that keeps the scheduler permanently non-active, and the (now repaired) test_live_hatchet_* live shape tests. Load when debugging CI runs, fixing validation.yml, pinning the Hatchet image, or planning Plan J live-worker validation.

UMD CI & Hatchet Deployment Reality

Mental Model

CI is a 5-job GitHub Actions workflow (.github/workflows/validation.yml): Ruff lint, Mypy strict, Unit (no DB, -m "not postgres"), PostgreSQL integration (postgres:17 service container, full suite), Docker E2E (native compose, no DinD/socket). The Hatchet worker is a GATED candidate subsystem: SDK 1.38.1 + server v0.105.2 are pinned but the production API still runs on the in-process DurableDAGRunner, so /v1/capabilities NEVER reports an active scheduler and the public-boundary E2E skips. The compose pin ghcr.io/hatchet-dev/hatchet is WRONG — the real public images live at sub-paths.

Coverage

Documented: run 33164294061 job conclusions; all 14 postgres-job failure root causes; working-tree-uncommitted fixes; docker-e2e step-6 failure; ghcr.io image-path reality; SDK/server release pairing; API runner wiring; defective live tests; follow-up verification 2026-08-28 (live manifest probes for all 5 sub-path images at v0.105.2; SDK 1.38.1 ClientConfig env surface incl. HATCHET_CLIENT_HOST_PORT + JWT-only token; lite vs split topology env requirements; cli.py :7070 host_port hardcode vs lite 7077; validate_hatchet_live.sh URL/port assumptions; hatchet.py:56 wrong image path; working-tree fix safety verdict). Not yet documented: exact v0.105.2-tag env surface (keyset-file vs SERVER_ENCRYPTION_MASTER_KEY; SERVER_MSGQUEUE_KIND=postgres validity) — must be read from the v0.105.2 tag docs/compose, not current docs. Documented (netns repair, 2026-08-29): runs 33228084721/33228376245 classification (true daemon race vs workflow-induced false absence vs optional-sandbox conflation); compose ps/run/up/profile/restart semantics; reconcile redesign facts. See the Netns section below. Documented (live Hatchet blocker, 2026-08-29): decisive run 33229130339/job 99038602321 (commit 6614b32) reached full split topology + readiness + real worker registration + external HTTP + live submissions, then EVERY live task failed before UMD execution — root cause verified against SDK py/1.38.1 primary source (task.py/hatchet.py/worker.py/workflow.py): task callbacks are invoked as fn(workflow_input, ctx) with the DIRECT submitted input dict (no v0 {"input": ...} wrapper); UMD _make_handler (hatchet.py:229) is a one-arg handler reading payload['input']['manifest'] (:230) → TypeError on every dispatch (retries=0 → instant FAIL, zero stage_run rows). Separate defects: factory.start swallows decorator exceptions (hatchet.py:426 contextlib.suppress), cli.py:123 readiness count falls back to len(work_registry) (fabricated readiness), engine-visible registration test (test_hatchet_live.py:1035-1080) inspects only local Standalone objects, hermetic _invoke_callback (test_hatchet_live.py:459-464) + hermetic connectivity (:1393) encode the same wrong v0 wrapper. See the Live Blocker section below. Last extended: 2026-08-29 (live Hatchet blocker primary-source verification)

Key Findings

CI run 33164294061 job results (GitHub API, repo xiaden/Universeity)

• Ruff lint: PASS; Mypy strict: PASS; Unit (no DB): FAIL; PostgreSQL integration: FAIL (14 failed / 550 passed / 17 skipped); Docker E2E: FAIL at step 6 ("Build API/worker image and start pinned Compose stack", 36s — immediate, not the 240s worker gate).

All 14 postgres-job failures are environment/package defects — NOT product-assertion failures

• 7× python-multipart missing: test_api_contract.py::test_multipart_upload_ingest_and_job_completes, ::test_multipart_missing_file_422; test_phase4_heterogeneous_ingestion.py::test_text/image/audio/subtitle_through_public_api. Root cause: python-multipart==0.0.32 is in the WORKING TREE pyproject.toml + deploy/pins/runtime.txt but was NOT in the committed tree at a6b1a62 (qa-test-analyzer L25 flagged this pre-push). src/umd/api/routers/sources.py calls request.form() for the /v1/sources multipart branch.
• 5× ffmpeg missing: test_phase4_heterogeneous_ingestion.py::test_video_composes_scenes_subtitles_audio_baseline; test_production_media_branches.py::test_production_registry_video_branch_commits_scenes_and_subtitles, ::..._honest_asr_gate_when_unavailable; test_video_subtitle_integration.py::test_video_inventory_and_audio_branch, ::test_embedded_subtitle_tracks_extracted_and_evidence, ::test_dialogue_video_audio_reaches_asr_and_subtitles_independent. Hosted runner has no ffmpeg; committed validation.yml has no install step (working tree adds one).
• 1× pg_dump missing: test_separation_ownership.py::test_postgres_backup_restore_boundary — hardcoded /usr/lib/postgresql/17/bin/pg_dump absent on runner; working tree adds conftest _resolve_pg_bin() (UMD_PG_BIN > which(pg_dump) > fallback) + postgresql-client-17 install.
• 1× compose interpolation: test_deployment_phaseE.py::test_docker_compose_config_when_daemon_present — HATCHET_COOKIE_SECRET/HATCHET_MASTER_KEY unset in test env (${VAR:?} in compose.yaml); working tree adds env.setdefault. Also fails in the Unit job (not postgres-marked).

docker-e2e step-6 failure = WRONG ghcr.io image path (GENUINE deployment gap)

• deploy/compose.yaml:104 pins ghcr.io/hatchet-dev/hatchet:${HATCHET_VERSION:-v0.105.2} — the TOP-LEVEL image is 403 DENIED / nonexistent on ghcr.io (verified via token+manifest probes; astral-sh/uv control gets a token, hatchet-dev/hatchet does not).
• The REAL public images (all 200 OK, v0.105.2 tag exists): ghcr.io/hatchet-dev/hatchet/hatchet-engine:v0.105.2, hatchet-lite, hatchet-admin, hatchet-migrate (per official docs.hatchet.run/self-hosting/docker-compose).
• SDK/server pairing is REAL: hatchet-sdk==1.38.1 on PyPI (2026-08-25, requires-python <4,>=3.10); server v0.105.2 is a real GitHub release (2026-08-25T15:54:16Z, "Hatchet 0.105.2" binary assets); py/1.38.1 tag exists in the same repo. Both released same day — contemporaneous pair, but still unproven end-to-end (no live cluster test yet).

API still wires DurableDAGRunner — boundary E2E skips even when compose is up

• src/umd/api/app.py:167-168 wires DurableDAGRunner(executor=..., store=job_store) — the interim in-process runner, NOT ProductionDAGRunner/Hatchet.
• src/umd/jobs/capability.py: scheduler status ∈ {gated, configured-but-unavailable, disabled, reference-only, active}; NEVER active without a live reachable cluster (lines 13-17, 66-79).
• tests/test_api_boundary_e2e.py _require_production_path (lines ~113-136) SKIPS unless /v1/capabilities reports an active scheduler/worker — so even a green docker-e2e cannot prove the production worker path today. Exec-manager log L82 note (a) confirms: rewire to ProductionDAGRunner is REQUIRED before release evidence.
• Handoff HATCHET_LIVE_VALIDATION_HANDOFF.md §6: the three test_live_hatchet_* shape tests are DEFECTIVE (use _RecordingClient + executor=None → no stage_run rows, callbacks_bound=False); must be repaired (bound real executor + real SDK client) or excluded honestly; primary release evidence = test_boundary_restart_duplicate_retry_and_consistency.

Handoff §6 DEFECT REPORT is STALE — the 3 live shape tests are ALREADY repaired

• Current committed tests/test_hatchet_live.py (unmodified in the working tree) builds a REAL DurableStageExecutor via _build_executor(umd_db) (lines 70-89), a REAL SDK client via _real_client() (~291-310: ClientConfig(token=..., host_port=<url-host>:7070) default 7070), and polls Postgres with _poll_until for stage_run/StageCompleted (lines 920-1006). The handoff §6 "RecordingClient + executor=None cannot pass" report describes the Plan I P4-S1 state and is now FALSE on disk.
• What remains unproven is LIVE EXECUTION: no run has ever executed against a real cluster, so SDK-surface mismatches (task-name namespacing, run_workflow payload shape, gRPC host_port routing) are untested. The honest gate is "run live and fix what surfaces", NOT "rewrite the tests". The CI-repair adversarial log (T1, 2026-08-28) makes the same two corrections.

CapabilityReporter has NO connectivity probe (active is unreachable by construction)

• src/umd/jobs/capability.py _scheduler_report() (lines 46-84) only checks find_spec("hatchet_sdk") + env presence; with env+SDK it returns configured-but-unavailable (lines 75-84). The docstring claims "a reachable client is the only thing that flips it to active" but NO reachability check exists in code. Consequence: even after rewiring app.py to ProductionDAGRunner against a fully live stack, /v1/capabilities never reports active → test_api_boundary_e2e _require_production_path still skips. A real connectivity probe (gRPC admin health / admin roundtrip) is a REQUIRED repair.

Compose worker service cannot configure OR run (two independent gaps)

• deploy/compose.yaml:81-99 worker service env block has ONLY UMD_ROLE + the umd-api-env anchor — NO UMD_HATCHET_SERVER_URL/UMD_HATCHET_TOKEN/HATCHET_CLIENT_HOST_PORT. .env.example has no UMD_HATCHET_* entries either, so env_file adds nothing. Worker exits 2 "not configured" even when the live gate is ON.
• deploy/Dockerfile runs pip install . (base deps only); hatchet-sdk lives in the optional worker extra (pyproject.toml:79-81). The worker container image therefore LACKS hatchet_sdk → cli.worker() exits 2 "hatchet_sdk not installed". Dockerfile (or a worker build stage) must install the worker extra.

Compose hatchet service is non-functional even after the image-path fix

• deploy/compose.yaml:101-111 sets only SERVER_AUTH_COOKIE_SECRET/SERVER_ENCRYPTION_MASTER_KEY; NO DATABASE_URL/msgqueue env, NO hatchet-migrate + hatchet-admin quickstart config bootstrap, NO published ports, NO config volume. Neither the split topology (migrate→admin→engine+dashboard) nor hatchet-lite (needs DATABASE_URL, grpc 7077, dashboard 8888, /config volume) can start from this service as written. Official compose: engine maps 7077:7070 and dashboard 8080:80; lite dashboard 8888 + grpc 7077; token provisioning via hatchet-admin token create --config /hatchet/config --tenant-id <uuid> (docs.hatchet.run/self-hosting/docker-compose, checked 2026-08-28 via Context7).

Uncommitted validation.yml converts the live worker gate to OPT-IN

• Working-tree diff: docker-e2e job gains UMD_VALIDATE_LIVE_WORKER: "${UMD_VALIDATE_LIVE_WORKER:-false}"; "Build API image and start pinned Compose stack (db + api)" (up db api only unless true); "Wait for worker/scheduler readiness (gate, opt-in)" with if: env.UMD_VALIDATE_LIVE_WORKER == 'true'. This is EXACTLY the anti-pattern the CI-repair DD lists ("weakening the live worker gate by opt-in or excluding worker/hatchet from default Docker E2E"). It is an honest deferral (committed gate could never pass) but per R4/R6/R7 the repair must RESTORE a fail-closed gate with a working topology, not ship the opt-in default.

Stale docs claims (contradict implemented behavior; Plan J P3 must fix)

• README.md:9-11 + 161-165 claim public routes pass an empty work registry to SynchronousRunner which "records requested stages as complete without invoking modality work" — FALSE: SynchronousRunner (src/umd/api/runner.py) is dead code, zero production imports (verified by grep), banned by test_production_architecture.py; production runs DurableDAGRunner over the real StageWorkRegistryFactory registry.
• README.md:90 "469 passed, 4 skipped" stale (Plan G QA R3 measured 555/3/14; CI 550/14/17); README.md:120 "migrations 0001–0006" stale (0007 exists); docs/providers.md:44 Hatchet row says pin v0.50.0 (stale — candidate v0.105.2) and "in-process runner is the local/job facade" (stale).

R&D workflow state for the CI repair (R8) — partially complete

• Done (2026-08-28): CI-repair DD stub (problem statement + R1-R6 constraints + anti-patterns, no architecture/decision), adversarial log T1 ideator (approaches A Commit-and-Wire top pick, B Split-Job, C Prove-Then-Run, D Lite; technology validation table), rnd-architect options report.
• NOT done: adversarial T2-T8 (counter-ideator/improver/counter-improver), complexity advisor, estimator, DDAuthor (final DD), PatternEnforcer. rnd-counter-ideator/improver logs dated 2026-08-25 belong to the MAIN DD process, not the CI-repair.

Netns race / reconcile facts (2026-08-29, runs 33228084721 + 33228376245)

• R16 (33228084721, job 99035605497) — TRUE transient daemon race present: Error response from daemon: bind-mount /proc/<pid>/ns/net -> /var/run/docker/netns/<id>: no such file or directory during compose up (random PID/netns IDs, daemon origin = moby netns lifecycle race on hosted runners; per support-debugger L4). PLUS three real defects (exec-manager L103): (1) worker gRPC TLS mismatch — engine SERVER_GRPC_INSECURE=t plaintext but hatchet_sdk 1.38.1 defaults tls_config.strategy=tls → WRONG_VERSION_NUMBER after readiness line, worker never registered (fixed b3e3127 via HATCHET_CLIENT_TLS_STRATEGY=insecure); (2) reconcile docker compose ps -q (no -a) EXCLUDES exited one-shots → hatchet-migrate/admin always absent though compose-ps proved Exited(0) (fixed e6b5c3f via ps -a -q); (3) sandbox-runner seccomp bwrap EPERM (statx/fsmount).
• R17 (33228376245, job 99036443345) — NO daemon netns error; full startup succeeded (preflight, db healthy, migrate/admin Exited(0), JWT minted, worker emitted C6 readiness + "waiting for tasks" — genuine registration). Reconcile failed on sandbox-runner=restarting (getcwd/vfork EPERM) + hatchet-migrate=absent hatchet-admin=absent (the ps -q visibility bug) across 5 blind retries. Failure = workflow-induced false absence + optional-service conflation, NOT a race.
• Optional sandbox conflation (compose.yaml:223-251 vs validation.yml:328/347): sandbox-runner is profiles: ["sandbox"], declared "OPTIONAL / profile-gated / OPT-IN" with restart: unless-stopped, yet the workflow FORCES --profile sandbox AND lists sandbox-runner in required_running. A sandbox failure therefore blocks the mandatory Hatchet/API gate. sandbox-runner env has NO UMD_HATCHET_TOKEN/UMD_HATCHET_SERVER_URL/HATCHET_CLIENT_HOST_PORT (compose lines 236-242 only set UMD_ROLE/UMD_SANDBOX_PROFILE + anchor) → its command: ["worker"] hits cli.worker() exit-2 gate (unconfigured) → with restart: unless-stopped it crash-loops even after the seccomp fix. Prediction to verify: next run with moby-default seccomp still shows sandbox-runner=restarting (exit 2), unless env is added or sandbox is excluded from required_running.
• Authoritative compose semantics (docker/compose + docker/docs, Context7 2026-08-29): ps shows ONLY running containers by default; -a/--all adds stopped incl. run-created one-offs; ps -q on multiple matches prints multiple IDs (multi-line breaks single-state compare). removes the one-off container on exit and OVERRIDES the service restart policy; skips dependency startup. builds/(re)creates/starts and restarts exited containers (desired state = running). Targeting a profiled service by NAME implicitly activates its profile (E2E-verified). ∈ created/running/paused/restarting/removing/exited/dead; bool + + via are the observable crash-loop signals. Restart never restarts on exit 0.

Live Hatchet blocker — v1 callback contract (VERIFIED against SDK py/1.38.1 source, 2026-08-29)

• SDK invocation contract (task.py @ py/1.38.1): Task.call(ctx) and aio_call(ctx) both run self._fn(workflow_input, cast(Context, ctx), **dependencies) — TWO positional args, input FIRST, context SECOND. workflow_input = self._workflow._get_workflow_input(ctx) = input_validator.validate_python(ctx._workflow_input) (workflow.py) — the DIRECT submitted input dict (EmptyModel extra=allow), NOT a v0 {"input": ...} wrapper. DependencyFunc protocol is def __call__(self, input, ctx, ...). Primary source: raw.githubusercontent.com/hatchet-dev/hatchet/py/1.38.1/sdks/python/hatchet_sdk/runnables/task.py + workflow.py.
• Correct UMD v1 handler: def handler(input, ctx) reading input["manifest"] → StageManifest.from_dict(...). Submitted v1 shape (runner.py:232-245): {"job_id", "source_id", "dag_universe", "stage", "manifest": manifest.to_dict(), ["causation_id"]} — verified by test_hermetic_submit_to_callback_connectivity asserting sub["input"]["manifest"].
• Affected callers: production hatchet.py:229 handler(payload) (registered :405 recording path, :428 real-SDK path); tests encoding the wrong wrapper: _invoke_callback test_hatchet_live.py:459-464 cb({"input": {"manifest": ...}}), hermetic connectivity :1393 cb({"input": sub["input"]), and their users :476 :491 :1424 :1439 :1442. cluster+docker live-gate tests affected: test_callback_executes_through_durable_executor (:467-480), test_callback_persists_stage_job_audit_transitions (:483-494).
• durable_task EXISTS in 1.38.1 (hatchet.py @ py/1.38.1): hatchet.durable_task(*, name=None, retries=0, ..., eviction_policy=DEFAULT_DURABLE_TASK_EVICTION_POLICY, ...) -> Standalone; handler typed (TWorkflowInput, DurableContext); sets is_durable=True + durable slot via Task.init/to_proto. Sync durable fns only get a deprecation warning (async-only in future) — still functional. hatchet.task/durable_task/workflow only CONSTRUCT local Standalone objects — ZERO engine contact.
• Engine-visible registration = Worker.start() → register_workflows() → admin.put_workflow(wf.to_proto()) (worker.py @ py/1.38.1), then the action-listener subprocess registers worker_id + opens gRPC streams. Hatchet.worker(name, workflows=...) type-checks (Standalone passes) and derives slot_config. So "engine-visible" proof REQUIRES a started/registered worker or engine-DB rows — never local Standalone inspection.

worker_ready_line honesty invariant

• src/umd/jobs/hatchet.py worker_ready_line returns EXACT string worker ready: registered {N} Hatchet workflows (candidate, pending Plan J live validation); cli.py prints it with flush=True BEFORE the blocking worker.start() (SDK 1.38.1 Worker.start() runs run_forever() and never returns); test_no_fake_gated_ready_claim scans cli.py for bare "worker ready". Wait script greps "worker ready: registered". Never remove the "(candidate, pending Plan J live validation)" suffix while unproven.

Follow-up Verification (2026-08-28)

Live image manifests at v0.105.2 — all sub-paths 200 OK

• Probed ghcr.io/v2/hatchet-dev/hatchet//manifests/v0.105.2: hatchet-lite 200, hatchet-lite-dev 200, hatchet-engine 200, hatchet-admin 200, hatchet-migrate 200. Only the top-level hatchet-dev/hatchet path is denied. hatchet-lite-dev exists (auth compiled out, fixed worker token) — the CI-friendliest option.

Topology facts (docs.hatchet.run/self-hosting/docker-compose + /self-hosting/hatchet-lite)

• Split (production): hatchet-migrate (needs DATABASE_URL, runs first) → hatchet-admin quickstart (generates /hatchet/config + certs; env: DATABASE_URL, SERVER_MSGQUEUE_RABBITMQ_URL or SERVER_MSGQUEUE_KIND=postgres, SERVER_AUTH_COOKIE_, SERVER_GRPC_, SERVER_INTERNAL_CLIENT_INTERNAL_GRPC_BROADCAST_ADDRESS) → hatchet-engine (grpc 7070 default; --config /hatchet/config) + hatchet-dashboard (8080). RabbitMQ optional.
• hatchet-lite (single container): needs DATABASE_URL (Postgres as DB+msgqueue by default), SERVER_GRPC_BIND_ADDRESS=0.0.0.0, SERVER_GRPC_INSECURE=t, SERVER_GRPC_PORT=7077, SERVER_URL, SERVER_AUTH_COOKIE_*; dashboard 8888, grpc 7077; /config volume; docs example does NOT use SERVER_AUTH_COOKIE_SECRET/SERVER_ENCRYPTION_MASTER_KEY (UMD compose's required ${VAR:?} vars come from the split-admin flow, not lite).

SDK 1.38.1 config surface (sdks/python/hatchet_sdk/config.py + client.py @ py/1.38.1)

• ClientConfig(BaseSettings) with env_prefix HATCHET_CLIENT_ → env var HATCHET_CLIENT_HOST_PORT is the correct override name (cli.py:105 checks it — correct).
• DEFAULT_HOST_PORT = "localhost:7070".
• Token REQUIRED and must be a valid JWT (must start with ey; config.py:263-272; tenant_id is derived from the JWT). umd-ci-token (validation.yml default) is NOT a JWT → ClientConfig raises → worker can never register. The live gate needs a real token (dev-image fixed token or hatchet-admin token create).
• When host_port is not explicitly set, the SDK derives it from the token's grpc_broadcast_address JWT claim (config.py:287-288) — but cli.py:104-106 ALWAYS overrides host_port to <url-host>:7070 unless HATCHET_CLIENT_HOST_PORT is set, so the JWT-derived address is bypassed.
• Hatchet.__init__(config=ClientConfig), self.admin, self.runs (RunsClient with admin_client()) all exist — matches umd.jobs.hatchet.py's runs.admin_client().run_workflow submission path.

UMD code gaps (concrete, must be coordinated)

• cli.py:104-106 hardcodes host_port=<url-hostname>:7070 → breaks hatchet-lite (grpc 7077 per docs example) and breaks split topology when the URL host is the dashboard (not the engine). Fix options: set HATCHET_CLIENT_HOST_PORT=<engine-host>:<port> in compose worker env, or configure lite with SERVER_GRPC_PORT=7070, or drop the override and rely on the JWT broadcast claim (requires correct SERVER_GRPC_BROADCAST_ADDRESS at token-issue time).
• validate_hatchet_live.sh:29 default UMD_HATCHET_SERVER_URL=http://hatchet:8080 assumes dashboard-on-8080 topology; lite dashboard is 8888; split dashboard is a separate service. URL hostname feeds cli.py's host_port derivation, so URL choice and topology must agree.
• src/umd/jobs/hatchet.py:56 HATCHET_SERVER_IMAGE still the WRONG top-level path and is surfaced as server_image by /v1/capabilities — must change together with deploy/compose.yaml:104 (P1-S3 pin test enforces cross-surface agreement).
• Current compose hatchet service env (only SERVER_AUTH_COOKIE_SECRET/SERVER_ENCRYPTION_MASTER_KEY, no DATABASE_URL, no msgqueue, no config step) matches NEITHER topology — the service is non-functional even after the image-path fix.

Working-tree fix safety verdict

• python-multipart==0.0.32 VERIFIED real (PyPI 2026-06-04, py3-none-any wheel, requires-python >=3.10, not yanked, Apache-2.0) — safe pin, fixes 7 failures.
• validation.yml ffmpeg + PGDG noble-pgdg postgresql-client-17 + UMD_PG_BIN via GITHUB_ENV: sound and correctly ordered (env set before the test step; pg_dump 17 ↔ postgres:17 service match).
• conftest _resolve_pg_bin(): sound — Path(pg_dump).resolve().parent follows the alternatives symlink to the versioned dir.
• test_deployment_phaseE env.setdefault for HATCHET_COOKIE_SECRET/HATCHET_MASTER_KEY: sound, mirrors the CI workflow env, interpolation-only.
• docker-e2e restructure (db+api default; worker gate opt-in UMD_VALIDATE_LIVE_WORKER): honest deferral, NOT weakening of a passing gate (step 6 always failed pre-fix). CAVEAT: the opt-in live gate is guaranteed-fail if enabled today — compose.yaml:104 image can't be pulled and UMD_HATCHET_TOKEN=umd-ci-token isn't a JWT.

Hosted Docker netns race (Plan K R10/R11/R15) — daemon-environmental, NOT app defect

• Exact error: Error response from daemon: bind-mount /proc/<pid>/ns/net -> /var/run/docker/netns/<id>: no such file or directory during docker compose up -d full-stack start (10 services). Root cause class = Docker daemon libnetwork/osl namespace-lifecycle race (moby/moby#50750, containerd/containerd#12141, moby/moby#46490, langgenius/dify#7739 — identical error on ordinary compose up in unrelated projects).
• Mechanism: moby >=28.0 removed the OCI prestart hook; networking is now configured between containerd task-create and task-start using the returned pid (moby/moby#51077; daemon/start.go initializeCreatedTask). If /proc/<pid>/ns/net vanishes in that window (shim exit/GC/restart), the bind-mount fails. GitHub ubuntu-24.04 runners ship Docker Client/Server 28.0.4+ (matrix-org/complement#775) — so this window exists on every hosted run.
• Trigger amplifier: simultaneous container creation/start (~10 at once). Docker Compose parallel ops are a known flake source (docker/compose#12747 concurrent map writes, fixed 2.36.0; runner-images#12160). Maintainer-blessed workaround: COMPOSE_PARALLEL_LIMIT=1 (serializes compose create/start) — directly narrows the race window, preserves topology and gates.
• R15 symptom (up exits 0 while 4 services stay Created): daemon-side start abort is NOT propagated as a pipeline failure → the committed post-up state reconciliation (2d84d07; ps -a fix b3e3127/e6b5c3f) with docker compose ps -a + docker inspect --format '{{.State.Status}}' per service is the correct fail-closed deterministic gate. Never diagnose a Created-leftover as "compose up failed" via exit code alone.
• NOT an app/topology defect: compose uses default bridge (no custom networks/network_mode), error occurs before app code; identical error reproduced by unrelated projects. Do NOT fix by reducing mandated services, host networking, DinD, socket mounts, or Hatchet Lite (DD non-goals/rejected alternatives).

Sandbox-runner seccomp EPERM (R16/R17) — SEPARATE root cause from Docker netns

• R16: statx/fsmount EPERM during bwrap namespace setup; R17: getcwd/vfork EPERM + Cannot fork before the worker could fork — the provisional 15-syscall sandbox profile could not host the full worker runtime. Worker (Docker default seccomp) booted+registered fine in R17 — Docker's default surface is the runtime precedent.
• Fix (committed e6b5c3f, P3-S3 amendment parallel-lavender-rabbit): sandbox-seccomp.json = moby v28.0.0 default allowlist (414 syscalls) + pivot_root only (415 total), defaultAction SCMP_ACT_ERRNO/errno 1 retained, cap_drop ALL/read_only/no-new-privileges/tmpfs retained, no privileged. Verified in-tree: 415 syscalls, all bwrap-critical syscalls present.
• Residual risk (watch R18+): bwrap --unshare-all inside a cap_drop ALL container on Ubuntu 24.04 may still hit AppArmor unprivileged-userns restrictions (kernel.apparmor_restrict_unprivileged_userns). If sandbox-runner still crash-loops post-moby-profile, that is a seccomp/security-posture DESIGN question (bounded options: per-service apparmor/seccomp unconfined for sandbox-runner only, or documented honest gated OS-isolation with SubprocessSandboxRunner bounded-failure containment — Task.md §32 allows "where practical"; DD/Plan K Phase 5 lists sandbox isolation as a permitted-gate status). NOT a topology change, NOT a reason to skip sandbox or remove namespaces.

R18 (33228898244, e6b5c3f) — FIRST GREEN TOPOLOGY; remaining blockers are host toolchain + test isolation

• Topology step PASSED for the first time: split topology up: db api worker sandbox-runner hatchet-engine hatchet-dashboard running; one-shots hatchet-migrate hatchet-admin exited. The moby-default seccomp surface (e6b5c3f) + ps -a reconcile fix (e6b5c3f) are both VALIDATED. sandbox-runner boots to running under the 415-syscall profile — the residual AppArmor userns risk flagged for R18 did NOT manifest.
• Netns race still fires (~1 hit in the topology step) and the reconcile recovers it fail-closed. The bounded retry remains DEAD CODE: the topology step has NO set -o pipefail (other steps do, lines 417/442/472), so docker compose ... 2>&1 | tee /tmp/compose-up.log always exits 0 (tee) → compose_up_ok=1; break → retry branch unreachable (R11/R16 logs confirm no execution-time "transient Docker netns race detected" line). The reconcile loop is the ONLY effective recovery — this is why R18 passed despite the race. Recommend adding set -o pipefail (or PIPESTATUS[0]) to the topology step so non-netns up failures also surface immediately; reconcile stays as the deterministic gate.
• Chain: /v1/ready PASS, wait-for-worker PASS (worker READY (registered workflows bound with Hatchet)), external probes PASS. Live suite FAILED with 4 errors, ALL environment/test-side, NOT product:
  1. test_live_hatchet_duplicate_and_restart_preserve_single_completion — ModuleNotFoundError: No module named 'hatchet_sdk' at test_hatchet_live.py:303 (_real_client()). Host runner installs only pip install -e ".[dev]"; hatchet-sdk lives in the worker extra (pyproject worker=[hatchet-sdk==1.38.1]). Working-tree fix (uncommitted): pip install --quiet --no-cache-dir -e ".[worker]" at the top of the live-suite step. 2-4. test_live_hatchet_retry_and_quarantine_single_authoritative_completion, test_live_hatchet_universe_change_drains_and_rekeys, test_live_hatchet_engine_visible_registration_exact_umd_stages — psycopg.errors.UniqueViolation: duplicate key value violates unique constraint "source_pkey", Key (id)=(b3f98f72-0000-0000-0000-000000000000) — _ensure_source() does a plain src_t.insert() of the fixed _SOURCE_ID against the SESSION-scoped shared live_db; the second test to seed collides. Working-tree fix (uncommitted): pg_insert(src_t).values(...).on_conflict_do_nothing(index_elements=["id"]) in tests/test_hatchet_live.py.

Stale artifacts found during RCA (2026-08-29)

• HATCHET_LIVE_VALIDATION_HANDOFF.md §6 (lines 184-210) still describes the 3 live tests as _RecordingClient/executor=None — ALREADY repaired in-tree (see above).
• HATCHET_LIVE_VALIDATION_HANDOFF.md §1 still shows the top-level denied image; actual src/umd/jobs/hatchet.py:61 HATCHET_SERVER_IMAGE = ghcr.io/hatchet-dev/hatchet/hatchet-engine:v0.105.2 (fixed P2-S2).
• Plan K (TASK-universal-media-decomposer-K-ci-repair-release-gate.md) P2-S2/P2-S4 "Done" notes OVERCLAIM: they state the 2-arg handler + engine-visible exact umd- registration are "proven by the live tests" — FALSE (handler is 1-arg; registration test is local-only). The live-blocker amendment must correct those notes.
• CONTRACTS.md:62 "readiness requires a live client and real callback registration" overstates WorkerHandle._ready (callbacks_bound only, no live contact) — implementation must be brought up to the contract.
• The netns DD amendment (artifacts/designs/pending/DD-universal-media-decomposer-plan-k-netns-workflow-amendment.md, Proposed, uncommitted) ALREADY contracts AT-16 (v1 callback + hermetic alignment + real-SDK negative test), AT-17 (tenant eligibility), AT-18 (durable_task + v1_task.is_durable=true) mapped to Plan K P2-S4/P2-S5/P3-S3. The live-blocker DD must CONTINUE/ground those (run 33229130339/job 99038602321) and ADD the independent defects above — it must NOT duplicate the netns bash/retry/reconcile contracts.

Critical Invariants

• Do not ship UMD_VALIDATE_LIVE_WORKER opt-in as the final posture — the live worker gate must be fail-closed release evidence with a working topology (image path, worker env, SDK install, real JWT token), per CI-repair DD anti-patterns.
• Do NOT add a capability "connectivity probe" that returns active without a real gRPC/admin roundtrip — active must require a genuinely reachable cluster (CONTRACTS.md:63).
• The 3 live shape tests are already repaired (real client+executor) — never re-report them as RecordingClient/executor=None defects; the open proof obligation is live execution.
• Do not weaken the HONESTY CONTRACT: no unconditional skips, no stubs replacing real execution, no claiming gated behavior as active; every skip must be a named permitted gate with CI proof (Plan J Problem Statement + DoD matrix rules).
• Worker container env (UMD_HATCHET_SERVER_URL/UMD_HATCHET_TOKEN/HATCHET_CLIENT_HOST_PORT) and Dockerfile worker-extra install are REQUIRED compose/Dockerfile changes; the current worker service/image cannot start under any topology.
• Historical suite must run unmodified/unweakened in CI — the workflow only selects/runs it.
• The hatchet image pin fix must use the correct sub-path (hatchet-engine or hatchet-lite), not the nonexistent top-level path; both SDK and server pins must stay lockstep-bumped with a new DAG universe + drain per handoff upgrade rule.
• Docker E2E must not fake worker readiness; the worker gate only counts when the worker genuinely registers with a reachable cluster.
• UMD_HATCHET_TOKEN must be a real JWT (ey-prefix) for any live run; the umd-ci-token placeholder can never register.
• cli.py host_port, compose worker env (HATCHET_CLIENT_HOST_PORT) and the chosen server topology (lite 7077 vs split engine 7070/dashboard 8080) must be changed together — a URL/port mismatch yields a worker that cannot connect.
• compose.yaml:104 and src/umd/jobs/hatchet.py:56 (HATCHET_SERVER_IMAGE) must be fixed together; the P1-S3 pin test enforces cross-surface agreement and /v1/capabilities surfaces server_image.

Sources

• CI: run 33164294061 via GitHub Actions API (job+step conclusions); /tmp/postgres-ci.log (test-postgres full output)
• Files: .github/workflows/validation.yml (committed a6b1a62 vs working tree), deploy/compose.yaml, deploy/Dockerfile, deploy/pins/runtime.txt, pyproject.toml, src/umd/api/app.py, src/umd/jobs/capability.py, src/umd/jobs/hatchet.py, src/umd/deploy/cli.py, tests/test_api_boundary_e2e.py, tests/test_hatchet_live.py, tests/conftest.py, tests/test_deployment_phaseE.py, tests/test_separation_ownership.py
• External: ghcr.io token+manifest probes; pypi.org/pypi/hatchet-sdk/1.38.1/json; api.github.com/repos/hatchet-dev/hatchet/releases/tags/v0.105.2; docs.hatchet.run/self-hosting/docker-compose
• Artifacts: artifacts/designs/parts/universal-media-decomposer/HATCHET_LIVE_VALIDATION_HANDOFF.md; artifacts/plans/pending/TASK-universal-media-decomposer-J-api-boundary-ci-release.md
• Logs: support-researcher L3, L6 (netns RCA); exec-manager L96-L104 (R10-R17 routing); support-debugger L4 (netns/seccomp RCA); exec-manager L101 (R15 Created-leftover), L103 (R16 TLS/one-shot/seccomp), L104 (R17 registration milestone + amendment)