odu

name

odu

description

Reference for the `odu` runner — how to invoke a full pipeline, a single recipe, or a platform-pinned node, and how to attach to a live run, from a project whose CI odu runs. Trigger when the user asks to "run CI", "run the pipeline", "re-run a check", to run named lanes or recipes (e.g. "run fmt and nix", "just the e2e lane", bare selectors like `fmt`/`nix`/`e2e`), or names a recipe by `<recipe>@<platform>`. This skill — not a repo's local `just ci` / `just <recipe>` — is how an odu-run request is served.

odu

odu (Tamil ஓடு — "run") runs the just recipe DAG tagged [metadata("ci")] across platforms and posts GitHub commit statuses per <recipe>@<platform> context. Unlike batch runners, the run is live state you attach to: the coordinator serves a typed surface on .ci/odu.sock, so status/logs/attach are in-band — no process-compose, no separately-versioned socket client.

A request to run CI is a request to run odu — never just ci. Many consuming repos expose a just ci (or just <recipe>) target that runs a pipeline locally. Do not shell out to it: it is a parallel, non-attachable path that bypasses everything odu gives you — the live surface, per-node GitHub statuses, structured results, fail-fast, cancel/supersede, and the log resources below. "run CI", "run fmt and nix", "re-run the e2e lane" all mean drive an odu run, by the MCP face first and the odu CLI otherwise.

Prefer the MCP face for runs. When the odu-mcp skill is present (the mcp__odu__* tools — check for an odu MCP server before shelling out), drive runs through it — run (pass selectors for named lanes/recipes) → wait_for_settle (fail-fast) → read the red node's log → node_rerun, with cancel / run({supersede}) to call off or replace a run. It spawns the same coordinator but gives you structured results and the fail-fast loop instead of scraping terminal output. The nix run … -- run CLI below is the reference and the fallback when no MCP server is wired.

Don't block on the full run — fail-fast is the point. wait_for_settle defaults to fail_fast: true: it returns the instant the first node goes red, while the slow lanes (e2e, build) keep running. That instant includes that node's LOG — a verdict is not published until the node's output is on disk — so when you drill in, the summary is already there rather than still on the wire. What you wait for is that one node's remaining backlog, not the rest of the DAG: a red fmt unblocks you while e2e is still running, and a node whose log ended first publishes with no wait at all. That early return is your unblock signal — drill into the red node's log and start fixing at once; never sit through the remaining lanes to "see the full status". The verdict is explicitly partial when it trips: fail_fast_tripped: true with settled: false, and failed[]/errored[] list only what's red so far — a floor, not the final tally, so more lanes may still fail. Conversely passed: true is the only trustworthy green — it comes solely from a fully settled run with zero red; never infer success from a fail-fast return. The coordinator does not stop when the tool returns: only your call did, so you can node_rerun the fixed node against the still-live run (the pending slow lanes keep it alive; linger covers the case where it already settled) and wait_for_settle again to catch any reds that surfaced meanwhile — or run({supersede}) when the fix is a new commit. Don't pad timeout_ms and wait: the loop is fail-fast → fix → re-wait, not one long block.

Retrying one lane is node_rerun, never supersede. node_rerun re-runs a single node (<recipe>@<platform>) and its dependents on the run that is still live, alongside the sibling lanes, cancelling nothing — the other platforms keep going and the run keeps its coordinator, venue leases and GitHub statuses. run({supersede}) cancels the WHOLE run, every lane of it, and is for replacing a run with a different commit. Superseding to retry a flaky linux lane throws away the darwin lane that was still running and green — the expensive operation for a job the cheap one does. Same rule on the CLI: odu rerun <selector>, not odu run --supersede.

A verdict names its run; no run fails loud. A verdict about an observed run carries that run's identity — sha7 always, and seq (sha7#seq) whenever the coordinator reserved an ordinal — so you match it to the run you dispatched, not a previously-settled one; pass expected_sha (a full sha or a sha7 prefix) to make that a hard, loud check. (seq is null only when none was reserved — a wait that saw no frame, or the rare case the coordinator couldn't reserve one; the run then claims sha7 but no unique sha7#seq.) And wait_for_settle never returns an empty nothing-verdict: called with no live run in the checkout it fails loud, not an instant settled: false — and WHICH loud it is answers the incident, not just the absence: a killed run leaves residue a clean exit removes (the run lock, the socket, the unfinished reservation), so when that residue is there the error names the death — "the run died with the process that started it", from the one deadRun read in @odu/run-client that runs (dead_run field), node_rerun, run, and the CLI twins all answer — and otherwise it mirrors odu status's "no run in progress". So a loud error means start or find a run (or read history with runs) — never hand-roll a process-liveness poll as a workaround.

Logs are a resource, not a tool. Don't look for a log-tail tool — there isn't one. A node's output is the MCP resource surface://collections/logs/{id} ({id} is the node, e.g. ci::unit@aarch64-darwin), read with ReadMcpResourceTool: the live buffered tail while the run is up, else the durable per-SHA log on disk. So when wait_for_settle returns a red node, the "read the log" step is ReadMcpResourceTool on that node's surface://collections/logs/{id} — subscribe for push updates, or just re-read to poll. (surface://streams/nodes is the pipeline snapshot resource alongside it.) These URIs resolve in the MCP server's HOME checkout only: for a run you targeted with a named checkout argument, read the log off THAT disk instead — join(checkout, logPathFor(sha7, id)) from @odu/run-client, never the home URI (which shows another tree's run, or "missing").

Invoking

nix run github:juspay/odu -- <subcommand> [args]

Pin a ref for reproducibility, or — if the consuming repo npins-pins odu and re-exports it (kolu does) — prefer its own flake output so the version is repo-controlled:

nix run .#odu -- <subcommand> [args]

Modes

Strict by default — odu run refuses a dirty tree, pins HEAD via git worktree, posts commit statuses, and splits per-recipe logs into .ci/<sha>/<plat>/<recipe>.log. Three flags relax that policy:

Flags	Tree	HEAD pin	Status posts	Use for
(none — default)	clean (refuses dirty)	git worktree at HEAD	posted	"real" CI runs
--no-post	clean	git worktree at HEAD	none	non-GitHub strict consumers; debugging strict without writing the PR's check list
--no-snapshot (implies --no-post)	live working tree	none	none	strict-mode dev iteration without clean-tree refuse
--no-strict (meta — same as --no-snapshot --no-post)	live working tree	none	none	dev iteration; the one-flag opt-out for "just run the pipeline"

Every mode ends with the same ── ci run summary @ <sha7> ── verdict block (the sha reads <sha7>+dirty for a live-tree run on uncommitted changes) and exits non-zero if any node failed or errored.

Common invocations

# Full pipeline (the [metadata("ci")] root, every configured platform).
nix run github:juspay/odu -- run

# Dev iteration on a dirty tree: no clean-tree refuse, no HEAD pin, no posts.
nix run github:juspay/odu -- run --no-strict

# Re-run a single failed recipe on one lane — overwrites the same GitHub
# commit-status context the full run wrote (closes the red check).
nix run github:juspay/odu -- run e2e@x86_64-linux

# One recipe across every pipeline platform; selectors compose.
nix run github:juspay/odu -- run e2e lint

# Restrict the WHOLE fanout to one platform (repeatable).
nix run github:juspay/odu -- run --platform x86_64-linux

# Skip the dependency closure; run ONLY the named nodes (_ci-setup still rides).
nix run github:juspay/odu -- run --no-deps e2e@aarch64-darwin

# A different DAG root instead of the [metadata("ci")] recipe.
nix run github:juspay/odu -- run --root ci::e2e

# One-shot redirect of a platform's host (pins one box; waits if busy).
nix run github:juspay/odu -- run --host x86_64-linux=my-build-box

# Fail immediately when every host in a pool is busy (default: wait in line).
nix run github:juspay/odu -- run --no-wait

# Venue inventory — free / busy / held-by for every configured host.
nix run github:juspay/odu -- hosts

# Agent-held lease across runs (no re-queue between odu run iterations).
nix run github:juspay/odu -- lease
nix run github:juspay/odu -- lease x86_64-linux --no-wait
nix run github:juspay/odu -- release

# One NDJSON line per node transition, for agents/tools driving CI:
# {"node":"ci::e2e@x86_64-linux","recipe":"ci::e2e","platform":"x86_64-linux",
#  "status":"running|success|failed|skipped|errored","exit_code":1,
#  "log":".ci/<sha7>/x86_64-linux/ci::e2e.log"}
nix run github:juspay/odu -- run --progress json

Without --progress json, output adapts to where stdout points: a live colour lane-matrix with a log-tail footer on a TTY; quiet transition lines plus a once-a-minute "… still running" heartbeat when piped.

Inspection subcommands (no side effects)

nix run github:juspay/odu -- dump            # resolved pipeline as JSON
nix run github:juspay/odu -- graph           # dependency graph (Mermaid)
nix run github:juspay/odu -- protect --dry-run   # the (recipe × platform) contexts
nix run github:juspay/odu -- protect             # require exactly those on the branch
nix run github:juspay/odu -- protect --create    # …making the ruleset if absent
# --platform P (repeatable) pins the repo's platform set with no hosts config;
# omitted, the set derives from the machine's hosts file (warned on stderr).
# The contexts land in the GitHub ruleset governing the branch; classic branch
# protection is not written. Without a ruleset protect refuses — `--create`
# makes one holding only the required checks, enforcing, exempting nobody.

Live introspection (attach to a run in progress)

While odu run is live in a checkout, these attach to its surface over .ci/odu.sock:

nix run github:juspay/odu -- status          # snapshot; -o json → {nodes, posting, run}
                                             # (warns while GitHub posts are owed;
                                             #  `run` = {phase, elapsed_ms, lanes[]}
                                             #  lane = {state: claiming|leased, …})
nix run github:juspay/odu -- attach          # live TUI dashboard on a tty
                                             # (digits attach · n/p cycle ·
                                             #  r rerun · q quit); -o json
                                             # = transition stream
nix run github:juspay/odu -- logs -f e2e@x86_64-linux   # -f returns once that
                                             # node's log is complete
nix run github:juspay/odu -- wait            # fail-fast JSON verdict (MCP wait_for_settle)
nix run github:juspay/odu -- wait --settle   # block until the whole run settles
nix run github:juspay/odu -- wait --expected-sha SHA [--timeout-ms N]
nix run github:juspay/odu -- rerun ci::unit@x86_64-linux   # one node
nix run github:juspay/odu -- rerun @x86_64-linux           # recipe nodes on that lane (not _ci-setup)
nix run github:juspay/odu -- rerun unit                    # that recipe on every lane
nix run github:juspay/odu -- cancel          # stop the live run, cleanly
nix run github:juspay/odu -- cancel @aarch64-darwin   # drop one platform lane
nix run github:juspay/odu -- cancel ci::fmt@x86_64-linux  # cancel one node
nix run github:juspay/odu -- runs            # this CHECKOUT's history (flags unposted
                                             # statuses); per-user view: `history list`

No run in progress ⇒ exit non-zero with no run in progress in this checkout (no live socket at .ci/odu.sock). One run per checkout — a second odu run refuses while the socket is live.

A run is attachable before it has lanes. The socket comes up before the venue claim, so status / attach / logs -f / wait all see a run from the moment it exists — including the minutes a cold host spends receiving the runner closure, which used to read as "no run in progress". In that window status prints a provisioning <elapsed> block naming the pool each lane is claiming from (run.phase is provisioning under -o json), _ci-setup@<platform> is running with the copy's own copying path … narration in its log (logs -f _ci-setup@x86_64-linux), and wait blocks instead of refusing. A claim that never succeeds lands as a red _ci-setup@<platform> with the reason in its log — a verdict and a runs record, not a vanished socket.

A red node's log holds the whole recipe, summary and all. That is the point of drilling into it, so a node's VERDICT waits for its output: a terminal status is not published until that node's log has ended, sealed in the same breath the durable file is. By the time anything tells you a node went red — wait_for_settle, odu wait, the commit status, the runs record, the settle verdict itself — the summary is already on disk. That holds on every path, --linger included, where the coordinator never tears down at all: the promise is kept where it is made rather than on the way out. The join is needed because a node's status arrives on a different stream than its output and gets there first, and a recipe's final lines — the N scenarios (2 failed) that says what went wrong — are the last to land. The same holds for the durable file and for logs -f. A lane that goes silent still owing output, or a run stopped before a node finished (cancel, an interrupt, the --linger idle self-reap), stamps [odu] log truncated: … into the log rather than ending mid-line, so a short log is never mistaken for a quiet recipe. And because the file is addressed by commit, not by run, re-running the same SHA REPLACES .ci/<sha>/<plat>/<recipe>.log — you are never reading two runs concatenated.

Wait / rerun (plain-CLI agent loop). odu wait is the CLI twin of MCP wait_for_settle — the same settle core, over a reader built the same way, so the two faces answer a run alike: default fail-fast (return the instant a node goes red), --settle for the full run; prints one JSON verdict line; exit 0 only on a fully-settled all-green run. A wait rides out a dropped LINK: if the connection to the coordinator dies while the run is still going (a busy coordinator can go quiet long enough for the keep-alive to give up), the wait re-dials and keeps waiting rather than reporting a live run as unsettled. It returns when the run settles, a node goes red, timeout_ms elapses (and says timed_out), or the coordinator is genuinely gone — in which case the verdict comes from that run's finalized record on disk. odu rerun <selector> is the headless face of surface node.rerun (and of the attach TUI's r key) — restart node(s) on the still-live run by fan-in id, @platform, or bare recipe name.

odu rerun and MCP node_rerun are the SAME operation, and they are the answer to "one lane failed": the run stays up, the sibling lanes keep running, and nothing is cancelled. odu run --supersede / run({supersede}) is the other thing — it kills the whole live run, every lane — so it belongs to "test the fixed commit", not to "retry that lane".

Cancel / supersede / linger. Bare odu cancel drives the live run's teardown from a second process (finalize posted statuses, close lanes, drop the socket) and waits until it's gone — no need to wait out a doomed run or pkill the coordinator. odu cancel <node> or odu cancel @<platform> cancels only that node or lane (cancelled status, not red) and leaves the rest of the run settling — MCP twins node_cancel / lane_cancel. odu run --supersede cancels whatever's live here first, then starts ("stop this, run the fixed commit"). By default a run exits the instant it drains; odu run --linger keeps it serving past settle so a node can be rerun later (retry a flake), self-reaping after an idle period or on cancel.

Durable introspection (after the coordinator is gone)

Every run is also written to a per-user catalog — ODU_STATE_DIR, else $XDG_STATE_HOME/odu/runs (~/.local/state/odu/runs) on Linux, ~/Library/Application Support/odu/runs on macOS. It survives the checkout, so these answer after the coordinator exited, from another terminal, and after a git worktree remove. Nothing was retired: .ci is still written, odu runs is still the CHECKOUT view, and every live command above is unchanged.

--run R is R = a run id, a unique prefix of one, <sha7>#<seq>, or latest.

nix run github:juspay/odu -- history list [--all] [--limit N] [-o json]
                                             # the catalog, newest first (--all = every checkout)
nix run github:juspay/odu -- history show --run R [--after CURSOR] [-o json]
                                             # one run's attention payload, without waiting
nix run github:juspay/odu -- history import [--dry-run] [-o json]
                                             # bring this checkout's .ci records in
nix run github:juspay/odu -- history prune [--days N] [--dry-run] [-o json]
                                             # expire finished runs past the window (30d default)
nix run github:juspay/odu -- logs --run R [--attempt N] [--offset B] [--limit B] [-o json] <node>
                                             # ONE recorded attempt; byte offsets, not lines
nix run github:juspay/odu -- wait --run R [--after CURSOR] [--deadline-ms N] [--settle] [-o json]
                                             # bounded, resumable; exits below
nix run github:juspay/odu -- rerun --run R [--request-id ID] [--expect-attempt N] [-o json] <selector>
                                             # retry a RECORDED run

The loop: start → bounded wait → diagnose → retry → resume. odu wait --run returns on the first ACTIONABLE red (a failure whose log has had its last word), not on settle — a fast lane's failure is reported without waiting out the slow ones; --settle waits for the whole run. Its exits are the contract, because "there is something to fix", "nothing has happened yet" and "its coordinator died" need different next moves: 0 passed · 1 a failure to act on — which does NOT mean settled, a red lane beside a still-running one already counts · 2 still going, nothing red at the deadline (ask again with the returned cursor) · 3 owner lost — provably gone without finalizing, start a fresh run · 4 no such run, or expired · 5 request refused (e.g. a cursor belonging to another run; the refusal carries a resync command). The bare odu wait above is UNCHANGED at 0/1. --after CURSOR resumes without showing you the same events twice, and the cursor advances only through events actually delivered.

Evidence is per ATTEMPT and old attempts are immutable — a retry adds N+1 and never overwrites the log you are reading (the .ci file, addressed by commit, still does). odu logs --run reports complete as a FIELD, so a truncated log says so rather than reading as a quiet recipe. odu rerun --run retries a recorded run and odu picks what that means: a new attempt if its coordinator is still up, else a NEW run linked to it, replayed from recorded inputs with the commit pinned (and not posting — a selection's verdict is not the pipeline's). A dirty live-tree run cannot be replayed and is refused. --request-id makes a repeat safe: same id, same input replays the recorded answer instead of starting a second run. --expect-attempt N refuses if the node has moved on since you read it — checked by the coordinator as it accepts the reset, so the node cannot advance between the check and the mutation it guards. Expiry leaves a tombstone, so an old run id answers "it existed, it failed, its evidence aged out" rather than "no such run".

Hosts config

$ODU_HOSTS (a file path) → ~/.config/odu/hosts.json → fallback ~/.config/justci/hosts.json (zero-config migration from justci):

{
  "x86_64-linux": ["ci-1", "ci-2", "ci-3"],
  "aarch64-darwin": "[e-mail]"
}

Keys are Nix system tuples; values are anything ssh dials, a list of them (a venue pool), or localhost (runs directly against the snapshot, no closure copy). A plain string is a pool of one. For each platform, odu run picks a free machine and leases it for the run: the coordinator dials odu-runner (same agent as the lane) over surface-remote and calls lease.claim — flock is a Nix dep of odu-runner, held by the agent process. Releases on finish / agent death unless an agent-held lease (odu lease / MCP lease) already covers the platform — then run reuses that host and leaves the lock alone. Busy pool → wait in line (or --no-wait fails); the whole claim is watchable live (see "attachable before it has lanes" above). A cold host is bounded by going silent, not by total time — the pin's idle bound (ODU_LEASE_CLAIM_TIMEOUT_MS, 180s) re-arms on every line the dial narrates, so a first run against a fresh box is not killed for being slow whether it is copying, evaluating or building. A second, absolute ceiling (ODU_LEASE_PIN_CEILING_MS, 45m) no line can move catches the other shape: the surface-remote session's own backstop retries rather than giving up and announces each retry as a progress line, so an idle-only bound would never fire on a host that keeps talking without finishing. The timeout message names which bound fired and what it was doing (… timed out after 180000ms without progress (still copying the runner closure — N store paths so far, last …)). odu hosts probes via lease.probe. Platforms absent from an existing config silently drop from the fanout, but a run that resolves zero lanes — no file anywhere, no --host, no --platform — is refused, not defaulted to localhost (juspay/odu#46). --host PLAT=ADDR pins one box for the run; run on this machine on purpose with --host PLAT=localhost or a "PLAT": "localhost" entry. A pool must be pure-local or pure-remote: mixing localhost with remotes is refused when a run leases that platform, because a lease-exempt localhost reads as always-free and starves the busy remotes beside it (juspay/odu#54). A mixed pool for a platform the run never leases is nobody's business and does not refuse the run (juspay/odu#66).

A lane host needs only ssh + Nix + outbound https: the runner ships as a Nix closure (nix copy → realise on the host), and the source arrives by git fetch of the pushed SHA — remote lanes cannot test unpushed commits (no git-bundle transport; push first). The lane host's own nix is used on the runner's PATH (never a pinned client — version skew against the host daemon corrupts CA-derivation handling).

Semantics worth knowing

• Lanes are one-shot: a lane whose ssh link dies mid-run fails as errored (GitHub state error, Errored (<dur>) description); live state does not survive a runner restart — the per-SHA log files do.
• Skipped nodes post no status: an absent required context is what blocks the merge.
• The coordinator resolves the generic lane runner from odu's own flake, not the repo under test: nix eval $ODU_RUNNER_FLAKE#packages.<platform>.odu-runner.drvPath, where ODU_RUNNER_FLAKE is baked onto the odu wrapper from self.outPath at build time. A consuming repo no longer re-exports odu-runner. There is no override or fallback — the runner is the exact build that shipped the coordinator (they share an RPC contract); a binary built without the baked flake refuses to run.

When NOT to use this skill

• Questions about odu's internals or design history — read the README and the kolu Atlas note A CI runner you attach to.
• Project-specific CI operations (warm pools, host leases, banned flags) — that's the consuming repo's operational docs, layered on top of this reference.