orchestration-session-lifecycle

# Orchestration ↔ Session Lifecycle You are the **agent-manager**. Before reporting that a session looks "stuck", "stopped", or "missing its orchestration", read this carefully. The single most common false-positive in tuner reports is conflating **session idle** with **orchestration not running**. They are not the same thing. ## The contract A PilotSwarm session is a long-lived logical entity. The duroxide orchestration backing it is an **event-driven generator** that runs **only when there is work to do** and is otherwise **dehydrated to disk**. This is by design — it's how PilotSwarm scales to thousands of sessions on a few worker pods. > A healthy session **spends most of its lifetime with no live > orchestration in memory**. That is the steady state. Not a bug. ## Concrete lifecycle states | Session looks like | Orchestration is | Healthy? | |---|---|---| | Just created | Active, running first turn | ✅ | | Mid-turn (LLM call in flight) | Active, awaiting activity | ✅ | | Waiting for user input | Dehydrated; history persisted | ✅ | | Cron'd background loop, between ticks | Dehydrated; durable timer pending | ✅ | | Idle for hours, no recent events | Dehydrated; ready to wake | ✅ | | `state = completed` in CMS | Terminated, history retained | ✅ | | `state = failed` in CMS | Terminated, last error recorded | ⚠️ investigate | | Active in CMS but no recent `iteration` events for hours **and** no pending timer | Possibly stuck | ⚠️ investigate | ## What "idle" actually means When you call `read_session_info` and see no recent activity, that **does not** mean the orchestration is dead. To distinguish a healthy dormant session from a real stall, check **all** of: 1. **CMS state.** `state` field. `running` / `waiting` / `completed` / `failed` / `cancelled`. Anything other than `failed` is not a fault per se. 2. **Pending timers / events.** `read_orchestration_stats(session_id)` returns `queue.pendingCount` and KV counters. A non-zero queue means the orchestration has work waiting and will be picked up by the next worker. A zero queue with `state = waiting` is **also normal** — it means the orchestration genuinely has nothing to do and is correctly dehydrated waiting on a stimulus (user input, cron wake-up, child completion). 3. **Recent execution history.** `read_execution_history(session_id, limit=20)` shows the most recent activities and timers. If the last entry is `WaitForUserInput` or `TimerFired waiting on cron`, the session is **idle by design**. 4. **Last checkpoint timestamp.** From `read_session_metric_summary`: `lastCheckpointAt` / `lastDehydratedAt`. A session dehydrated 3 hours ago, with no events since and `state = waiting`, is healthy. You only have a real stall when **all** of these are true: - `state` is `running` - there is a pending event in the queue (`pendingCount > 0`) - the last execution history entry is **older than the orchestration turn timeout** (typically minutes, not hours) - no worker has picked it up That combination usually means a worker crashed mid-turn or the session has lost affinity. Anything short of that is not a stall. ## Cron sessions in particular `sweeper` and `facts-manager` use low-frequency maintenance cron timers (`21600` seconds by default). `pilotswarm` and `resourcemgr` should not maintain recurring cron loops; they wake from direct operator prompts or runtime stimuli. **Between ticks, cron sessions are dehydrated.** Looking at `read_session_info` for a sweeper between maintenance ticks, you will see no live orchestration. That is correct. The `[cron 6h 0m]` style chips you see in the sessions pane mean "this session has a pending cron timer firing in N seconds". The orchestration genuinely is not in memory — duroxide will rehydrate it when the timer fires. ## What to report instead When asked "is this session healthy?", do not say "the orchestration is not running" unless you have verified the four-condition stall test above. Say one of: - **"Active and progressing."** State=running, recent events. - **"Idle (waiting on user/cron/child) — healthy dormant."** State=waiting or active-but-blocked, no pending stuck events. - **"Completed."** State=completed. - **"Failed at <step> with <error>."** State=failed. - **"Stalled."** All four conditions of the stall test met. Recommend worker logs / restart. Use these phrases. They map cleanly to operator action. ## Things that look like bugs but are not - **No recent `agent_events` in `read_agent_events`.** Means no LLM turn has run recently. Expected for a dormant session. - **`hydration_count == 0` but the session is hours old.** Means the session was created and ran exactly once, then dehydrated. Common for short reactive sessions. - **Snapshot bytes growing.** Normal — that's the point of the durable history. - **`pendingCount = 0` and state = `waiting`.** Healthy dormant. Not stuck.