analyze-trajectory

# Analyze Trajectory You are doing a **deep dive** into a recurring failure pattern. The harness's pre-computed `YOUR TRAJECTORY` block surfaces *that* something is recurring; this skill helps you understand *why* and produce a focused diagnosis. This skill exists because raw GitHub Actions logs are too large and noisy to digest in your main context window. The pattern (Recursive Language Model — see Reithan's reference in issue #226) is: keep your root context small, dispatch a sub-agent to read the raw logs, and have the sub-agent return a 1-3 sentence summary. Recurse if the summary surfaces a deeper question. ## When to use Trigger this skill when ANY of these hold: - `YOUR TRAJECTORY` flagged a `STUCK` task (≥3 attempts in window, 0 successes) - A CI error fingerprint appeared `≥2×` in the recurring-errors section - Multiple revert commits appeared across recent sessions (the trajectory's "Reverts in window" line shows the count) - A specific issue (e.g. `#205`) has been mentioned in multiple session journals without resolution ## When NOT to use - The trajectory looks healthy. Don't spelunk for problems that aren't there — that's just burning sub-agent budget. - The failure is well-understood already (you already know the cause from journal/learnings). Skip straight to the fix. - You're inside Phase B (implementation) and the failure is the task you're currently doing — fix it directly, don't recurse. ## Procedure ### 1. Frame the question (single sentence) Examples of well-framed questions: - *"Why does the evaluator phase fail with 'AnthropicError: rate_limit_exceeded' on sessions day-53, day-55, and day-56?"* - *"Why was the task 'Add /fallback flag' reverted on 6 separate sessions? What's the recurring blocker?"* - *"What does run 4321 look like at the moment of failure?"* A good question names a specific event (run id, session day, error fingerprint) and what you want to know about it. Don't ask vague questions like *"what's wrong with my trajectory?"* ### 2. Identify the artifact For each question, pick exactly one artifact to fetch: - **CI failure** → run id from the trajectory's CI errors section. `gh run view <id> --log-failed` (drop `--repo`; gh auto-detects from the local clone's origin remote, which is the right one) - **Reverted task** → commit SHA of the revert. `git show <sha>` and the next-newer commit's full diff - **Session-level wreckage** → audit.jsonl from that session. **Note**: `$YOYO_AUDIT_DIR` is set by the harness ONLY inside `scripts/skill_evolve.sh` (a different invocation than evolve.sh). When loaded inside a normal evolve session, you must fetch the audit-log branch yourself first: ```bash git fetch --depth 50 origin audit-log:audit-log AUDIT_WT=$(mktemp -d) git worktree add "$AUDIT_WT" audit-log ls "$AUDIT_WT/sessions/" | tail -10 # ... read what you need ... git worktree remove --force "$AUDIT_WT" ``` ### 3. Decide: direct read or sub-agent? Estimate the artifact size first: ```bash gh run view <id> --log-failed 2>/dev/null | wc -c ``` - **< 5KB**: read it directly with `read_file` or `bash`. Skip sub-agent — the cost isn't worth it. - **≥ 5KB**: dispatch a sub-agent. Don't load raw logs into your main context. ### 3.5. Handle large artifacts (token-aware chunking) Before dispatching a sub-agent, estimate whether the artifact fits in a single sub-agent's context: ``` estimated_tokens = artifact_bytes / 4 ``` **If estimated_tokens ≤ 30,000** (roughly half a sub-agent's context window): proceed to Step 4 as normal — single sub-agent dispatch. **If estimated_tokens > 30,000**: the artifact is too large for one sub-agent to digest reliably. Split and fan out: 1. **Split into chunks** of ~20,000 tokens (~80,000 bytes) with 2,000-token (~8,000 byte) overlap between consecutive chunks. The overlap ensures error context that spans a chunk boundary isn't lost. 2. **Store each chunk separately** in shared state: ``` shared_state set key="trajectory.run-<id>.chunk-1" value="<first 80KB>" shared_state set key="trajectory.run-<id>.chunk-2" value="<next 80KB, starting 8KB before the split>" ... ``` 3. **Dispatch one sub-agent per chunk** with the prompt: ``` You are analyzing CHUNK <N> of <M> from a CI log. The chunk is stored in shared state under key "trajectory.run-<id>.chunk-<N>". Read it with: shared_state get key="trajectory.run-<id>.chunk-<N>" Question: <your single-sentence question from step 1> Reply with ONLY a JSON object (no markdown fences, no prose): { "summary": "1-3 sentences on what this chunk reveals about the failure", "key_lines": ["relevant line 1", "relevant line 2"], "chunk_relevant": true, "confidence": "high|medium|low" } If this chunk contains no information relevant to the question, set chunk_relevant to false and keep summary/key_lines minimal. ``` 4. **Merge chunk results** — after all chunk sub-agents return, store their combined results in shared state and dispatch one final merge sub-agent: ``` shared_state set key="trajectory.run-<id>.chunk-results" value="<JSON array of chunk responses>" ``` Merge sub-agent prompt: ``` You are merging analyses from <M> chunks of a single CI log. The chunk analyses are stored in shared state under key "trajectory.run-<id>.chunk-results". Read them with: shared_state get key="trajectory.run-<id>.chunk-results" Original question: <your single-sentence question from step 1> Synthesize the chunk analyses into a single diagnosis. Reply with ONLY a JSON object (no markdown fences, no prose): { "summary": "1-3 sentences explaining the root cause", "key_lines": ["most important line 1", "most important line 2"], "deeper_question": null, "confidence": "high|medium|low" } ``` 5. The merge sub-agent's response is your diagnosis — validate it using the same JSON contract rules in Step 4. **Chunking counts toward the recursion cap** (Step 5): each chunk sub-agent is depth 1, the merge sub-agent is depth 1. If chunking used 4 chunk agents + 1 merge agent, you've used 1 of your 3 recursion levels. You can still recurse on a `deeper_question` from the merge result, but be mindful of the budget. ### 4. Dispatch a sub-agent (if needed) **Store the artifact in shared state first** — don't paste large logs into the sub-agent prompt. Sub-agents automatically have access to the `shared_state` tool and share the same key-value store as their parent. Use the namespace convention `trajectory.<key>` for all artifacts stored by this skill. ```bash # 1. Fetch the artifact into a shell variable LOG=$(gh run view <id> --log-failed 2>/dev/null) # 2. Store it in shared state (the parent agent calls this directly) shared_state set key="trajectory.run-<id>" value="$LOG" ``` Then dispatch the sub-agent with a **reference**, not the artifact itself: ``` Question: <your single-sentence question from step 1> The CI log is stored in shared state under key "trajectory.run-<id>". Read it with: shared_state get key="trajectory.run-<id>" Reply with ONLY a JSON object (no markdown fences, no prose) matching this schema: { "summary": "1-3 sentences explaining the root cause, with no surrounding quotes", "key_lines": ["file.rs:42:11 borrow of moved value", "AnthropicError: rate_limit_exceeded"], "deeper_question": null, "confidence": "medium" } Field rules: - summary: free string, 1-3 sentences - key_lines: array of 1-5 short strings (max 100 chars each) that prove the cause - deeper_question: JSON null when no follow-up is needed; otherwise a single-sentence string - confidence: exactly one of "high", "medium", or "low" ``` Sub-agents inherit RTK compression on bash output and directory restrictions, but they do NOT inherit skills. Keep the sub-agent prompt fully self-contained — don't reference other skills. Sub-agents share the parent's `SharedState` store automatically (via `SharedStateTool` wired by `build_sub_agent_tool`). **Validate the sub-agent response** — after the sub-agent returns, check: 1. **Parse as JSON.** Strip any leading/trailing whitespace and markdown code fences (` ```json ... ``` `) that sub-agents sometimes add despite instructions. 2. **Check required fields.** The parsed object must contain all four keys: `summary` (string), `key_lines` (array of strings), `deeper_question` (string or null), `confidence` (one of `"high"`, `"medium"`, `"low"`). 3. **If valid** → proceed to Step 5 (recursion check). 4. **If invalid** → retry ONCE with this prompt: ``` Your previous response was not valid JSON or was missing required fields. Please respond with ONLY a JSON object (no markdown, no explanation): { "summary": "1-3 sentences explaining the root cause", "key_lines": ["key line 1", "key line 2"], "deeper_question": null, "confidence": "high|medium|low" } Required fields: summary (string), key_lines (array), deeper_question (string or null), confidence ("high"|"medium"|"low"). ``` 5. **If retry also fails** → fall back gracefully (see below). **Sub-agent failure fallback** — if the sub-agent (a) errors, (b) returns non-JSON twice (initial + retry), (c) returns truncated JSON that can't be repaired, or (d) is unavailable as a tool: 1. Append the raw response to `memory/learnings.jsonl` as a learning entry with `pattern_key: trajectory.subagent_malformed_response` so we can debug later. 2. Extract whatever text the sub-agent did return and treat it as the `summary` field. Construct a synthetic response: `{"summary": "<raw text, first 500 chars>", "key_lines": [], "deeper_question": null, "confidence": "low"}`. 3. If even the raw text is empty or the sub-agent errored entirely, downgrade to a direct read of the artifact: use `shared_state get key="trajectory.run-<id>"` to retrieve the stored log, then read the last 50-100 lines in your main context. 4. Produce a low-confidence diagnosis from what you can see directly. Skip recursion (no point — sub-agent path is broken). 5. Mark the diagnosis with `confidence: low (sub-agent unavailable)` so downstream decisions know to be cautious. ### 5. Recurse if the sub-agent returns `deeper_question` If `confidence` is `"low"` AND `deeper_question` is a non-null string (JSON null returns false on this check, but if you see the literal string `"null"` treat it as null too — that's a sub-agent bug worth logging), run another sub-agent dispatch with the narrower question. Reuse the same artifact; the sub-agent will focus differently. **Hard cap: recursion depth = 3.** That's: initial dispatch → 1st recursion → 2nd recursion. After that, accept whatever you have. The cap is informed by the recursive-LM literature ([RLM blog, alexzhang13.github.io/blog/2025/rlm/](https://alexzhang13.github.io/blog/2025/rlm/)) and prevents runaway agent costs. If you hit the cap without `confidence == "high"`, that's still a valid outcome — write the diagnosis with whatever clarity you have and flag it as "needs follow-up". ### 6. Aggregate to a single diagnosis Produce a 3-5 sentence diagnosis paragraph that includes: - **What recurs**: one-line summary of the pattern - **Root cause** (or best-guess): from the sub-agent's summary - **Evidence**: ≤3 specific lines or run IDs - **Suggested next attempt**: one concrete action (a different approach, a new task, or "log to learnings.jsonl and skip for now") Write the diagnosis somewhere durable: - If you're in a normal evolve session and this informed your task choice → cite it in the assessment doc - If you're investigating a specific issue → comment on the issue with the diagnosis - Always also append a `learnings.jsonl` entry. The `pattern_key` field (optional in the standard schema, see `skills/communicate/SKILL.md`) takes a kebab-case `<verb>.<object>` value — for trajectory-derived diagnoses, use `pattern_key: trajectory.<short-slug>` (e.g., `trajectory.fallback_provider_stuck`, `trajectory.evaluator_rate_limit`). This lets skill-evolve cluster recurring trajectory findings. ## Pitfalls - **Don't ask the sub-agent to make decisions.** It summarizes evidence; you decide what to do. Sub-agents in chained recursion can drift if asked to plan. - **Don't recurse on `confidence: high`.** The whole point is to stop early when you have a clear answer. - **Don't dump multiple artifacts to one sub-agent.** One artifact per dispatch keeps the sub-agent focused and the JSON output reliable. Store each artifact under a separate `trajectory.<key>` in shared state. - **Don't forget the recursion cap.** 3 is the hard limit. If you find yourself wanting depth 4, your initial question was probably too vague — go back to step 1. - **Skills do not chain.** Sub-agents don't load this skill or any other; you must include the question and shared-state key reference in the sub-agent's prompt directly. - **Don't run this skill inside Phase B (implementation).** That's task-execution time, not introspection time. Save the diagnosis for the next session's Phase A1 (assess). ## Verification A diagnosis is "good enough" when ALL of: - It names a concrete file/line/condition (not "something with the API") - It cites at least one specific run id or commit SHA - The suggested next attempt is *different* from what's already been tried (otherwise you'll just hit the same wall) - The total work used ≤3 sub-agent dispatches If the diagnosis fails any of these, recurse one more time (within the cap) or accept the partial result and document the open question in `learnings.jsonl`. ## What this skill deliberately does NOT do - **Does not modify code.** Diagnosis is the output. The actual fix is a normal task on a future evolve session — it's better to step away with the diagnosis written down and let the next session's planning agent decide whether to act on it. - **Does not auto-create issues.** If the diagnosis is worth filing, do it via `communicate` skill in the same session — but it's a separate decision, not part of this skill's procedure. - **Does not write to `audit-log` branch.** The branch is read-only from this skill's perspective.