durable-execution

# Durable Execution Your code will crash. Durable execution means: when it restarts, it picks up where it left off — not from the beginning. ``` Traditional: start → step1 → step2 → crash → lost Queue-based: start → step1 → step2 → crash → retry from top → duplicates Durable: start → step1 → step2 → crash → replay to step2 → continue from step3 ``` Everything else — frameworks, protocols, infrastructure — is implementation detail. --- ## Do You Need This? Score 1–5 on each dimension: | Dimension | 1 (low) | 5 (high) | |-----------|---------|----------| | **Failure cost** | Retry is inexpensive, no side effects | Retry causes duplicates, data loss, or revenue loss | | **Duration** | Milliseconds, single request | Hours/days/weeks, spans process lifetimes | | **Coordination** | Single service, single step | Multiple services, human gates, external callbacks | | **State complexity** | Stateless or simple key-value | Branching workflows, conditional logic, fan-out/fan-in | | Total | Recommendation | |-------|---------------| | 4–8 | Use a task queue with idempotent handlers | | 9–12 | Durabilize the critical path only | | 13–20 | Durable execution is the right primitive | **Quick check — you need this if any are true:** - A crash mid-workflow means a customer gets charged but never receives the product - Your workflow spans multiple HTTP requests or process lifetimes - You need human approval gates that pause for hours or days - Duplicate execution of a step causes real-world harm (double charges, duplicate emails) - You have multi-step processes across multiple services that must all succeed or all roll back --- ## Choose Your Path ``` Does your workflow need... ├─ Just crash recovery + idempotency? │ └─ BAKED IN: add checkpointing to your existing DB. Zero infra cost. │ See: references/BAKED-IN.md │ ├─ Fan-out parallelism, durable sleep, human-in-the-loop, │ or cross-service coordination? │ └─ RESONATE: single-binary server + tiny SDK. ~$5/mo on a VPS. │ See: references/RESONATE-QUICKSTART.md │ └─ Enterprise-scale with managed infrastructure? └─ Temporal Cloud ($520/mo+ at scale) or AWS Step Functions. But consider: do you actually need that complexity? ``` | | Baked In | Resonate | Temporal | |---|---|---|---| | **Infra cost** | $0 (your existing DB) | ~$5/mo (VPS) to ~$170/mo (1M tasks/day) | ~$520/mo (1M tasks/day) | | **Dependencies** | None | Single binary + SQLite | Cluster + multiple services | | **Serverless** | Yes (any runtime) | Yes (Lambda, Edge Functions) | No (requires always-on cluster) | | **Setup time** | Minutes | 5 minutes | Hours to days | | **Patterns** | Checkpoint, idempotency, outbox | All 5 patterns + distributed coordination | All patterns + enterprise features | | **Learning curve** | Low (just SQL + your code) | Low (sequential code — generators in TS/Py, or async/await in TS/Rust/Go) | High (proprietary DSL + concepts) | | **When it fits** | Single-service, sequential workflows | Multi-service, any complexity | Large teams with dedicated infra staff | --- ## Baked In — Framework-Free Durability Three building blocks. No framework. Just your database. ### 1. Idempotency Keys Every operation gets a deterministic ID. Before executing, check if it already ran. ```typescript async function runOnce<T>(db: Database, key: string, fn: () => Promise<T>): Promise<T> { const existing = db.query("SELECT result FROM completed_steps WHERE key = ?").get(key); if (existing) return JSON.parse(existing.result); const result = await fn(); db.run("INSERT INTO completed_steps (key, result) VALUES (?, ?)", [key, JSON.stringify(result)]); return result; } ``` ### 2. Step-Level Checkpointing Wrap each step. On crash and restart, completed steps return cached results. Execution resumes from the first incomplete step. ```typescript async function durableCheckout(db: Database, orderId: string) { const inventory = await runOnce(db, `${orderId}:reserve`, () => reserveInventory(orderId)); const payment = await runOnce(db, `${orderId}:charge`, () => chargeCard(orderId, inventory)); const shipment = await runOnce(db, `${orderId}:ship`, () => createShipment(orderId, payment)); const email = await runOnce(db, `${orderId}:notify`, () => sendConfirmation(orderId, shipment)); return { inventory, payment, shipment, email }; } ``` ### 3. Outbox Pattern Side effects (emails, webhooks, API calls) go to a table first. A separate process delivers them exactly once. ```typescript // Inside your workflow — write to outbox, don't send directly db.run("INSERT OR IGNORE INTO outbox (id, payload) VALUES (?, ?)", [`${orderId}:confirmation-email`, JSON.stringify({ to: email, subject: "Order confirmed" })]); // Separate delivery loop — idempotent, retryable const pending = db.query("SELECT * FROM outbox WHERE delivered_at IS NULL").all(); for (const msg of pending) { await deliver(msg); // your send logic db.run("UPDATE outbox SET delivered_at = ? WHERE id = ?", [Date.now(), msg.id]); } ``` **When baked-in hits its limits:** - You need a workflow to sleep for days without a process staying alive → need durable timers - You need to suspend and wait for a human to approve something → need external promise resolution - You need to fan out work across multiple services in parallel → need distributed coordination - You want the framework to handle retries, timeouts, and replay for you → use Resonate **Full implementation with templates:** See `references/BAKED-IN.md` and `assets/baked-in-checkpoint.ts`. --- ## With Resonate — Durable Execution Platform Resonate's open-source server is a single binary (Rust + SQLite, zero external deps) paired with a tiny SDK. Runs anywhere — VPS, serverless, edge functions. Costs ~$5/mo on a small VPS. Your code is an ordinary function with durable steps — a generator in TypeScript/Python, an `async fn`/func in Rust/Go. TypeScript also ships a second engine that writes the same thing as `async function`/`await` instead of `function*`/`yield*` (see below). Each durable step is a checkpoint. If the process crashes, the server re-dispatches the work to any available worker, which replays from the last checkpoint. > **Language note.** The examples below (and in this skill's references) are shown in **TypeScript**. The concepts are identical across all four Resonate SDKs; only the syntax differs. For concrete, idiomatic syntax in your language, see the per-SDK skills — `resonate-basic-durable-world-usage-{typescript,python,rust,go}` for the Context API, `resonate-async-await-engine-typescript` for TypeScript's `async`/`await` engine, and the matching `resonate-saga-pattern-*` / `resonate-recursive-fan-out-pattern-*` / `resonate-human-in-the-loop-pattern-*` skills (and `resonate-durable-sleep-scheduled-work-{typescript,rust,go}`) for the patterns shown here. ```typescript import { Resonate, type Context } from "@resonatehq/sdk"; const resonate = new Resonate({ url: "http://localhost:8001" }); resonate.register("processOrder", function* (ctx: Context, orderId: string) { const order = yield* ctx.run(fetchOrder, orderId); const payment = yield* ctx.run(chargeCard, order); const shipment = yield* ctx.run(createShipment, order); yield* ctx.run(sendConfirmation, order.email); return { payment, shipment }; }); await resonate.run("order-42", "processOrder", "order-42"); ``` Prefer `async`/`await` over generators? TypeScript's other engine (`@resonatehq/sdk/async`, v0.11+) writes the same workflow without `function*`: ```typescript import { Resonate } from "@resonatehq/sdk/async"; const resonate = new Resonate({ url: "http://localhost:8001" }); resonate.register("processOrder", async (ctx, orderId: string) => { const order = await ctx.run(fetchOrder, orderId); const payment = await ctx.run(chargeCard, order); const shipment = await ctx.run(createShipment, order); await ctx.run(sendConfirmation, order.email); return { payment, shipment }; }); const handle = await resonate.run("order-42", "processOrder", "order-42"); await handle.result(); ``` Both engines talk to the same server and give the same guarantees — pick whichever reads better in your codebase. Each step is checkpointed; crash recovery, retries, and replay are automatic. **Get running in 5 minutes:** See `references/RESONATE-QUICKSTART.md`. **All patterns with full code:** See `references/RESONATE-PATTERNS.md`. **SDK API reference (TypeScript):** See `references/RESONATE-SDK.md` (for Python/Rust/Go, see `resonate-basic-durable-world-usage-{python,rust,go}`). **Starter template (TypeScript):** Copy `assets/resonate-worker.ts` (the `assets/` templates are TypeScript; for other languages start from the per-SDK skill). --- ## The 5 Patterns Shown here in the generator engine; every pattern translates directly to the async/await engine — swap `function*`/`yield*` for `async function`/`await` (see `resonate-async-await-engine-typescript`). ### 1. Saga — Multi-step with compensating rollbacks Each step is checkpointed. On failure, compensate in reverse order. ```typescript function* orderSaga(ctx: Context, orderId: string) { const completed: string[] = []; try { yield* ctx.run(reserveInventory, orderId); completed.push("inventory"); yield* ctx.run(chargePayment, orderId); completed.push("payment"); yield* ctx.run(createShipment, orderId); completed.push("shipment"); return { status: "success" }; } catch (error) { for (const step of completed.reverse()) { yield* ctx.run(compensate, step, orderId); } return { status: "rolled-back", compensated: completed }; } } ``` **Use when:** Multiple services must all succeed or all roll back (payments, bookings, provisioning). ### 2. Fan-Out / Fan-In — Parallel work with aggregation Dispatch work to parallel workers via RPC. Each branch is independently durable. ```typescript function* batchProcess(ctx: Context, items: string[]) { const results: string[] = []; for (const item of items) { const result = yield* ctx.rpc( "processItem", item, ctx.options({ target: "poll://any@item-workers" }) ); results.push(result); } return results; } ``` **Use when:** Batch processing, parallel API calls, map-reduce workloads. ### 3. Human-in-the-Loop — Suspend for external signal Workflow suspends without holding resources. Resumes when a human (or webhook) resolves the promise. ```typescript function* approvalFlow(ctx: Context, orderId: string) { // No id argument — ctx.promise() derives a deterministic id from the call // tree. Read it back off the returned future to hand to the approver. const approval = yield* ctx.promise<string>({ timeout: 48 * 60 * 60 * 1000 // 48 hours }); yield* ctx.run(sendApprovalEmail, orderId, approval.id); const decision = yield* approval; if (decision === "approved") yield* ctx.run(processOrder, orderId); return decision; } ``` **Use when:** Approval workflows, manual review gates, external callbacks, payment confirmation. ### 4. Scheduled / Cron — Durable timers Sleep is durable. Process can die and restart — the timer still fires. ```typescript function* onboarding(ctx: Context, userId: string) { yield* ctx.run(sendEmail, userId, "Welcome!"); yield* ctx.sleep(24 * 60 * 60 * 1000); // 1 day — survives crashes yield* ctx.run(sendEmail, userId, "Getting started tips"); yield* ctx.sleep(6 * 24 * 60 * 60 * 1000); // 6 days yield* ctx.run(sendEmail, userId, "How are we doing?"); } ``` **Use when:** Drip campaigns, SLA reminders, retry delays, polling loops, recurring jobs. ### 5. Entity — Long-lived mutable state Each method call is a durable step on a persistent entity. ```typescript function* orderLifecycle(ctx: Context, orderId: string) { const order = yield* ctx.run(createOrder, orderId); yield* ctx.run(validateOrder, order); const payment = yield* ctx.run(processPayment, order); yield* ctx.run(fulfillOrder, order, payment); yield* ctx.run(notifyCustomer, order); return { orderId, status: "fulfilled" }; } ``` **Use when:** Lifecycle management, state machines, long-lived domain objects. --- ## The Hard Problems ### Versioning You deploy new code. Old executions are mid-flight. The replay now hits different code paths than what was recorded. **Solutions:** Version-tag your workflows. Drain in-flight executions before deploying breaking changes. Or design steps to be additive (new steps at the end, never remove or reorder existing ones). ### Side Effects at the Boundary You sent an email in step 3. Step 4 crashes. On replay, step 3 returns the stored result — but the email was already sent. **Solutions:** - **Outbox pattern** — write to a durable outbox; a separate process delivers exactly once - **Idempotency keys** — pass a deterministic key to external APIs so duplicates are no-ops - **Accept-and-compensate** — accept that duplicates happen; send a correction if needed ### Testing Durability How do you test that replay actually works? That your workflow survives a crash at every possible step? **Approaches:** - **Kill-and-resume test** — start workflow, kill the process mid-step, restart, verify it completes correctly - **Replay unit test** — capture an execution log, replay against new code, assert same result - **Transition tests** — enumerate all valid state transitions, verify each one produces correct output - **Deterministic simulation testing (DST)** — inject controlled randomness across thousands of runs, verify invariants hold **Full testing guide:** See `references/TESTING.md`. ### Observability You cannot `console.log` your way through replay. You need to see: what step am I on, what's pending, what failed, what's the state of each promise. **What to monitor:** Execution list (ID, status, duration), step timeline/waterfall, promise state graph, worker health, queue depth, error rates, retry storms. --- ## Efficiency — Why This Matters Durable execution has a reputation for being expensive and complex. It doesn't have to be.