durable-execution

name

durable-execution

description

Make agents survive crashes, timeouts, restarts, and human waits — using Temporal, Inngest, Restate, or LangGraph's checkpointer. Cover the Workflow + Activity pattern, pause/resume semantics, retry policies, and when to retrofit (answer: before your first long-running agent goes to production). Use whenever the user mentions long-running agents, multi-hour tasks, pause/resume, retry on failure, agent crashing mid-flight, state persistence, Temporal, Inngest, Restate, or asks how to handle reliability over hours/days.

Durable Execution for Agents

Stateless agents lose everything on crash. Production agents run for minutes to hours, call external services, wait on humans, retry transient failures. Without durable execution, every restart loses work, every flake costs tokens, every human wait pegs a process to memory.

By 2026, durable execution moved from "nice to have" to table stakes for any agent running longer than a few minutes.

The pattern (one diagram)

┌─────────────────────────────────────────────────┐
│  Workflow (deterministic, replayable)          │
│  - Agent loop control flow                      │
│  - State transitions                            │
│  - "If verified, then commit; else retry"       │
└──────────────────┬──────────────────────────────┘
                   │ invokes
                   ▼
┌─────────────────────────────────────────────────┐
│  Activities (non-deterministic, retryable)      │
│  - LLM calls                                    │
│  - Tool / API invocations                       │
│  - Database writes                              │
│  - Human approval waits                         │
└─────────────────────────────────────────────────┘

Workflow code is deterministic — same inputs always produce same control flow. It's replayed from an event log on crash recovery; no LLM calls run on replay, the recorded results are reused.

Activity code is non-deterministic and side-effecting — LLM calls, network calls, anything that can fail or vary. Activities are individually retryable with exponential backoff.

State lives in the event log. Crash mid-flight? On recovery, the workflow replays the log to reconstruct state, then continues from where it stopped.

When to retrofit (answer: before the first long-running agent ships)

Retrofitting durable execution is painful. The agent loop has to be restructured into the Workflow + Activity split. Plan for it before launch.

Triggers that force durable execution:

• Any agent loop expected to run > 60 seconds end-to-end
• Any agent that waits for human input (HITL)
• Any agent that performs multiple side-effecting actions where partial completion is bad
• Any agent that needs to survive deploys, restarts, host failures
• Any agent that needs scheduled / cron-style invocation
• Any multi-agent system where orchestrator + workers must coordinate reliably

If any of these apply to v1, build durable from the start.

Implementation choices

Reference implementation (paved road): for running many long-lived agents as deployed infrastructure — durable backlog, coordinated scheduling, a bounded runner, fleet health — the family's Runtime & fleet operations surface is AgenticOps (bun add github:Moai-Team-LLC/AgenticOps), via the reference-stack skill. Recommended default, not a requirement — the options below stand on their own.

Temporal — the industrial standard

• Strengths: mature, deep ecosystem, first-party integrations for OpenAI Agents SDK, Pydantic AI, Vercel AI SDK, mcp-agent. Activity overhead is ~10–50ms — negligible vs LLM latencies.
• Trade-offs: operational complexity (cluster to run), learning curve on the deterministic-workflow model
• Pick when: you're already operating production infrastructure, need maximum reliability, or have many agents to coordinate

Inngest — simpler operationally

• Strengths: serverless model, fits TypeScript/Vercel deployments naturally, simpler mental model than Temporal
• Trade-offs: less mature than Temporal, smaller ecosystem
• Pick when: TypeScript-heavy team, want managed service, agents are simpler

Restate — newer, focused

• Strengths: modern API, designed for serverless-first deployments, simpler than Temporal
• Trade-offs: newer; smaller community than Temporal/Inngest
• Pick when: greenfield TypeScript/Python project, want simplicity

LangGraph checkpointer (Postgres) — built-in if already on LangGraph

• Strengths: zero external orchestrator; pause/resume/time-travel out of the box; major reason for LangGraph adoption at Uber, Klarna, LinkedIn
• Trade-offs: ties you to LangGraph; less flexible than full Temporal for complex coordination
• Pick when: you've already committed to LangGraph and your reliability needs fit within its model

The deterministic-workflow discipline

This is the hardest concept for engineers new to durable execution. Workflow code is replayed from event log — it MUST produce the same control flow when re-run with the same recorded events.

Rules for workflow code:

• No I/O directly. All I/O happens through activities.
• No time.time(), random(), or non-deterministic builtins. Use the framework's deterministic equivalents (workflow.now(), workflow.uuid4()).
• No reading external state outside activities. Even reading the database goes through an activity.
• Branch on activity results, not on global state. The workflow's job is orchestration, not data access.

Activity code has no such constraints — it's normal application code with retries layered on top.

Pattern: human approval as a durable signal

A common HITL pattern with durable execution:

# Pseudocode — exact syntax varies by framework
@workflow
def agent_with_approval(input):
    plan = activity(propose_plan, input)
    
    if requires_approval(plan):
        # workflow pauses here — may sleep for hours
        approval = wait_for_signal("user_approval", timeout="24h")
        if not approval.approved:
            return activity(handle_rejection, approval.feedback)
    
    result = activity(execute_plan, plan)
    return result

The workflow can sleep for 24 hours waiting for approval without consuming resources. When the user clicks approve, the signal wakes the workflow exactly where it left off.

This is impossible without durable execution. With it, it's a few lines of code.

Retry policies

Activities can fail transiently. The framework retries automatically — but you control the policy:

Failure type	Policy
LLM rate limit (429)	Exponential backoff, max 5 retries, jitter
LLM 5xx	Exponential backoff, max 3 retries
Tool timeout	Linear backoff, max 2 retries
Permission denied	No retry — fail fast
Schema validation error	No retry — bug, alert
Idempotency-safe write	Aggressive retry — should be idempotent anyway
Non-idempotent write	Single attempt — failure must be handled by workflow

Define these explicitly per activity type. The default is usually wrong.

Cost and latency

Per-activity overhead in Temporal is ~10–50ms — invisible against multi-second LLM calls. Don't optimize prematurely.

Where cost matters:

• Event log storage. Long-running workflows accumulate large event histories. Plan for archiving / compaction.
• Activity invocation count. Each activity has overhead. Don't make every tiny operation an activity — group related work.
• Worker pool sizing. Activities run on worker pools; oversubscribe and you queue.

When NOT to use durable execution

• Sub-second interactive responses. A user typing in a chat doesn't need workflow-level durability. The interaction itself is the unit.
• One-shot deterministic transformations. A simple "summarize this document" doesn't need durability.
• Read-only operations. If nothing is being mutated, crash recovery is cheap — just restart.

The threshold is roughly: does the work span more than ~30 seconds OR involve multiple side-effecting steps OR wait on external actors?

Diagnosing failures in production

Common patterns when durability is misconfigured:

Symptom	Cause	Fix
Workflow stuck forever	Activity has no timeout	Set activity timeouts; use heartbeat for long activities
Workflow replays incorrectly	Non-deterministic code in workflow	Move I/O / randomness to activities
Same action executed twice	Activity not idempotent + retry	Make activity idempotent, OR don't retry, OR use idempotency keys
Event log explodes	Logging too much in workflow state	Push large data to external store, keep references in workflow
Worker can't keep up	Worker pool too small	Scale workers; check for blocking activities

Output of this skill

When the conversation completes, the user should have:

1. A clear answer to "do we need durable execution?" with the deciding criteria
2. If yes: choice between Temporal, Inngest, Restate, LangGraph checkpointer with justification
3. The Workflow + Activity split sketched for their agent
4. Retry policy per activity type
5. HITL signal pattern if humans are in the loop
6. Plan for event log retention and worker scaling