trace-delegation

# Trace Delegation Use this skill to reconstruct what the delegation engine actually did. Prefer durable evidence from journal events and tracker-visible records over inferred architecture diagrams. ## Current Implementation Map | Concern | Source | | --- | --- | | Engine, tracker, state transitions, verification/checkpoint gates | `crates/runtime/src/server/delegation/engine.rs` | | HTTP handlers for delegate/list/pause/resume | `crates/runtime/src/server/delegation/handlers.rs` | | Patterns, tiers, request/result types, aggregation | `crates/services/src/coordination.rs` | | Journal event builders and metadata fields | `crates/services/src/session_journal.rs` | | Parent/child messaging and broadcast groups | `crates/astra-messaging/src/` | | Runtime lifecycle delegation hints | `crates/runtime/src/server/server_loop_host.rs` | Do not use the old path `runtime/src/server/delegation_engine.rs`; the engine lives under `runtime/src/server/delegation/`. ## Journal Events These are the stable delegation events to group by `metadata.delegation_id`: | Event | Key Metadata | | --- | --- | | `DelegationStarted` | `delegation_id`, `parent_run_id`, `pattern`, `agent_ids`, `agent_count` | | `DelegationSubRunStarted` | `delegation_id`, `sub_run_id`, `parent_run_id`, `agent_id`, `status`, `depth`, `retry_of` | | `DelegationRetry` | `delegation_id`, `original_run_id`, `retry_run_id`, `agent_id`, `attempt`, `reason` | | `DelegationSubRunCompleted` | `delegation_id`, `sub_run_id`, `agent_id`, `status`, `error`, `output_preview` | | `DelegationCompleted` | `delegation_id`, `pattern`, `total_sub_runs`, `succeeded`, `failed`, `aggregated_status`, `aggregated_output_preview` | Important limitation: journal sub-run completion events do not include per-sub-run token counts. `DelegationResult` and HTTP `DelegationResponse` carry token totals, but a journal-only trace cannot reliably reconstruct token distribution. ## Locate Delegation Data ```bash python3 - <<'PY' import glob, json, os events = [] names = { "DelegationStarted", "DelegationSubRunStarted", "DelegationRetry", "DelegationSubRunCompleted", "DelegationCompleted", } for path in glob.glob(os.path.expanduser("~/.astra/sessions/*.jsonl")): with open(path, "r", encoding="utf-8", errors="replace") as fh: for line_no, line in enumerate(fh, 1): try: event = json.loads(line) except Exception: continue if event.get("type") in names: meta = event.get("metadata") or {} events.append((event.get("ts", ""), path, line_no, event.get("type"), meta)) for ts, path, line_no, typ, meta in events[-80:]: did = str(meta.get("delegation_id", ""))[:16] rid = meta.get("sub_run_id") or meta.get("parent_run_id") or meta.get("retry_run_id") or "" print(f"{ts[:19]} | {typ:27s} | {did:16s} | {rid} | {path}:{line_no}") PY ``` Resolve the target: | Target | Action | | --- | --- | | Delegation ID | Filter events where `metadata.delegation_id` matches | | Parent run ID | Find `DelegationStarted.metadata.parent_run_id`, then collect its delegation IDs | | Session ID | Read that session journal only | | `last` or omitted | Use the most recent `DelegationStarted` | ## Patterns And Aggregation Pattern names emitted by the engine: | Pattern | Journal string | Notes | | --- | --- | --- | | `CoordinationPattern::FanOut` | `fan_out` | Parallel agents, aggregated after all complete | | `Pipeline` | `pipeline` | Current engine executes through the sequential path using stage agent IDs | | `Sequential` | `sequential` | Ordered agents, optional stop-on-success | | `AdversarialReview` | `adversarial_review` | Producer/reviewer rounds with max rounds | | `Fork` | `fork` | N tasks sharing parent context, one user-tier agent, constrained recursion | Aggregation strategies in current code are `FirstSuccess`, `AllResults`, `Consensus`, and `LlmGuided`. Do not report `Merge` or `VoteOnBest`; those are stale names. ## Trace Workflow 1. Build a timeline from journal events sorted by timestamp and line number. 2. For each `DelegationStarted`, verify the expected pattern and agent set. 3. For each expected child, confirm a `DelegationSubRunStarted` event. Missing starts mean the engine registered a delegation but the sub-run did not enter `Running`. 4. For each started child, confirm terminal `DelegationSubRunCompleted`. Missing completion means pause, cancellation, crash, timeout, or unfinished state. 5. Fold `DelegationRetry` into the original child. A retry should mark the original as `verification_failed` and create a `retry_of` relationship on the retry run. 6. Confirm `DelegationCompleted` totals match the completed child set. A mismatch is a journal or aggregation bug. 7. If pause/resume is involved, inspect `pause_delegations_handler`, `resume_delegations_handler`, `pause_flags`, `cancel_tokens`, and `cleanup_delegation` in `engine.rs`. ## Failure Classification | Symptom | Check | | --- | --- | | Delegation never starts | Engine configured in `AppState`, handler ownership check, request validation | | No child runs | `record_sub_run`, `transition_state(..., Running)`, request allowlists, recursion depth | | Child cannot message parent | mailbox router registration and `MessageTarget::Parent` resolution | | Retry loop | `VerificationGate::verify`, `max_retries`, `DelegationRetry.reason` | | Pause/resume stuck | `pause_flags`, child loop pause boundaries, `resume_children_of` | | Cancellation leak | `cancel_tokens`, `cancel_children_of`, `cleanup_delegation` | | Wrong aggregate | `AggregationStrategy` and `aggregate_results` in `coordination.rs` | | Missing UI progress | `progress_broadcaster`, `DelegationProgress`, SSE event emission | ## Report Format ``` Delegation Trace: <delegation_id> Pattern: <pattern> Parent run: <parent_run_id> Expected agents/tasks: <n> Timeline: - <ts> started: agents=<...> - <ts> sub-run started: <run_id> agent=<agent_id> depth=<n> retry_of=<...> - <ts> retry: <old> -> <new> attempt=<n> reason=<...> - <ts> sub-run completed: <run_id> status=<status> - <ts> delegation completed: succeeded=<n> failed=<n> status=<status> Findings: - <only concrete gaps or mismatches, with file/event evidence> Limits: - <state any data not present in the journal, such as per-sub-run token distribution> ```