execution-pipeline

name	execution-pipeline
description	The end-to-end cell execution pipeline from MCP tool call through daemon to kernel and back. Use when debugging execution failures, understanding output timing, investigating why outputs are missing or stale, or modifying the execute/run-all flow. Covers required_heads, CellQueued, RuntimeStateDoc polling, output-sync grace, and output resolution.

Cell Execution Pipeline

Use this skill when debugging execution-related issues: cells that don't execute, outputs that don't appear, execution that times out, or MCP tools that return empty results. This traces the full path from tool invocation to resolved outputs.

The Five Stages

Stage 1: Causal Precondition (required_heads)

Before sending an execute request, the client captures the current Automerge heads of the notebook document:

let required_heads = handle.current_heads_hex()?;

These heads are attached to the request envelope. The daemon's wait_for_required_heads() defers processing until all listed change hashes exist in its copy of the notebook document (checked via get_change_by_hash — a ChangeGraph containment check).

Why this matters: Without required_heads, the daemon might execute against stale cell source — the client wrote "x = 1" but the daemon hasn't received that sync frame yet, so it executes the old "x = 0".

Timeout: 10 seconds. If heads don't arrive, the daemon logs a warning and processes anyway (graceful degradation).

Frontend optimization: Before capturing heads, the frontend calls flushSync() to push any pending source edits into the sync stream, minimizing the daemon-side wait.

Stage 2: Request Submission

The client sends an ExecuteCell or batch RunAllCells request:

let request = NotebookRequest::ExecuteCell {
    cell_id: cell_id.to_string(),
};
let response = handle.send_request_after_heads(request, required_heads).await;

send_request_after_heads wraps the request in a NotebookRequestEnvelope with the captured heads and sends it through the sync task.

Stage 3: Daemon Queuing

The daemon receives the request, waits for required heads (Stage 1), reads the cell source from its synced notebook document, and queues execution with the kernel:

1. Daemon reads source from its NotebookDoc (the CRDT, not the .ipynb file)
2. Creates a unique execution_id
3. Writes to RuntimeStateDoc: execution entry with status "queued", then "running" when the kernel starts
4. Responds with CellQueued { execution_id, position } immediately
5. Forwards code to the kernel via ZMQ execute_request

Key invariant: The daemon writes set_execution_done(eid, success) to RuntimeStateDoc ONLY AFTER all output manifests for that execution are committed. This ordering guarantee is what makes RuntimeStateDoc polling reliable.

Control-plane invariant: Kernel lifecycle signals (KernelIdle, ExecutionDone, CellError, KernelDied) are not output transport. They must not share bounded queues with stdout floods, display churn, or Output widget replay. If output work is pending, drain lifecycle/control signals first so interrupts and queue release remain responsive.

Output-widget replay: RuntimeStateDoc is the durable source of truth for captured Output widget outputs. The kernel-facing SendCommUpdate replay is best-effort output work on a bounded queue. IOPub output arms must use non-blocking enqueue/drop semantics for replay and must not .await a bounded work-channel send before they can observe later status messages.

Display updates: update_display_data messages with display_id are transient display churn. Coalesce them by display_id and commit only the latest pending update off the IOPub hot path. Flush pending display updates after KernelIdle and before ExecutionDone so terminal runtime state still means durable output state is available. The display-update committer's Notify is only a wake hint; the pending map is the source of truth, and each wake or priority flush drains all currently pending display IDs.

Stream-output committer: stdout/stderr chunks may be coalesced and periodic flushes may be dropped when pressure is high. The terminal buffer holds the latest rendered state. Ordering-sensitive boundaries, such as display/error output after a stream, use the stream committer's priority path and wait for the stream flush before clearing terminal state. ExecutionDone also uses the priority path so the final stream manifest is durable before the runtime state becomes terminal.

Stage 4: Terminal Wait (RuntimeStateDoc Polling)

The client polls RuntimeStateDoc for execution completion:

await_execution_terminal(handle, &execution_id, timeout, None).await

Phase 1 — Terminal status poll:

• Polls every 50ms
• Checks executions[eid].status for "done" or "error"
• Also watches for kernel-level failure (kernel.lifecycle == Error|Shutdown)
• Returns KernelFailed if the kernel dies while execution is pending

Phase 2 — Output-sync grace:

• After terminal status is reached, the output list might still be empty on the client's replica (sync lag)
• Polls every 10ms for up to 500ms (the grace period)
• Exits as soon as outputs appear

Why RuntimeStateDoc, not broadcasts: The ExecutionDone broadcast arrives over a separate channel and the client's Automerge replica may not have caught up on the final stream writes. The RuntimeStateDoc is authoritative — once status is "done", outputs are guaranteed to be in the same document.

Stage 5: Output Resolution

Outputs are inline manifest Maps in RuntimeStateDoc, containing ContentRef entries per MIME type:

let outputs = output_resolver::resolve_cell_outputs_for_llm(&output_manifests, ctx).await;

Resolution depends on MIME type:

• Text MIME (text/*, application/json, image/svg+xml): Inline string if ≤1KB, or fetch from blob store as UTF-8
• Binary MIME (image/png, audio/*, etc.): Always blob store. Frontend gets http:// URL. Python gets raw bytes.
• Widget output (application/vnd.jupyter.widget-view+json): References comm topology/output routing in RuntimeStateDoc; mutable widget values live in the paired CommsDoc

MCP execution paths use preview mode — output is truncated for LLM consumption. Agents that need full output call get_cell(full_output=true) separately.

Execution ID Lifecycle

Client sends ExecuteCell
  → Daemon returns CellQueued { execution_id: "exec-abc" }
  → RuntimeStateDoc: executions["exec-abc"] = { status: "queued" }
  → Kernel starts: executions["exec-abc"].status = "running"
  → Outputs arrive: executions["exec-abc"].outputs = [manifest1, ...]
  → Kernel done: executions["exec-abc"] = { status: "done", success: true }
  → Client reads outputs from executions["exec-abc"].outputs

The execution_id is the stable reference for one execution attempt. If the same cell is executed twice, each gets a different execution_id. Agents can pass execution_id to get_cell() to read outputs for a specific execution rather than the cell's current outputs.

Run-All Flow

run_all_cells follows the same pipeline but batched:

1. Captures required_heads once
2. Sends RunAllCells request with all cell IDs
3. Daemon returns AllCellsQueued { cell_execution_ids } — a map of cell_id → execution_id for every queued cell
4. Client polls each execution_id in parallel with a shared deadline
5. Returns per-cell results

Timeout: The shared deadline applies to the entire run, not per-cell. If one cell takes 90% of the budget, remaining cells get less time.

Common Failure Modes

"Outputs are empty"

1. Output-sync grace too short: The execution finished but outputs haven't synced yet. The 500ms grace usually suffices, but very large outputs (big DataFrames, many plots) may need more time.
2. execution_id mismatch: Reading outputs with the wrong execution_id or reading the cell's "current" outputs after re-execution replaced them.
3. Blob store unreachable: Binary outputs reference blobs. If the blob HTTP server is down, resolution fails silently.

"Cell didn't execute"

1. required_heads timeout: Daemon waited 10s for heads that never arrived. Check if the sync stream is healthy.
2. Kernel not ready: The kernel isn't started or is in error state. The daemon returns an error response, not CellQueued.
3. Trust gate: Untrusted notebooks may block execution pending approval.

"Execution timed out"

1. Long-running cell: The cell genuinely takes longer than the timeout (default varies by caller — MCP uses 120s).
2. Kernel hung: The kernel process is alive but not responding. Check kernel.lifecycle in RuntimeStateDoc.
3. Sync stall: Terminal status was written by the daemon but the client's sync stream stopped delivering frames. Check daemon logs for sync errors.

"Stale outputs from previous execution"

The execution completed but the outputs visible belong to an earlier run. This happens when:

1. Reading cell outputs without using the execution_id — the cell's "current" pointer may not have been updated yet
2. The RuntimeStateDoc sync hasn't delivered the latest writes

Fix: Always use execution_id from the CellQueued response to read outputs for a specific execution.

Execution Document Architecture

Execution primarily spans NotebookDoc and RuntimeStateDoc in a notebook room:

Document	What it holds for execution
NotebookDoc	Cell source code (what to execute)
RuntimeStateDoc	Execution lifecycle, outputs, kernel status
CommsDoc	Mutable widget values referenced by widget outputs, gated by RuntimeStateDoc topology

The required_heads gate ensures NotebookDoc is synced before execution starts. RuntimeStateDoc polling ensures outputs are available before the client reads them. CommsDoc sync matters when those outputs include live widgets. All document sync streams run concurrently on the same socket connection.