improve-agent

# Improve Agent Analyze past coding sessions to find recurring agent issues, then fix them by updating AGENTS.md, creating new skills, or improving code/infra. **Multi-harness.** Both pi (`~/.pi/agent/sessions/<mangled-cwd>/`) and Claude Code (`~/.claude/projects/<mangled-cwd>/`) are supported. `extract.py` translates Claude Code records into the pi schema on read, so every mode below works on either source. Default source is the harness you are running under; override with `--source`. **Multi-device.** With `ANDENKEN_SESSION_CORPUS` set (`~/.env.local`, the same variable andenken's indexer and session-recap read), the gathered corpus is scanned *in addition to* the live stores — the corpus keeps each runtime's path shape under `<corpus>/<device>/`, so the other machine's sessions for the same project are found by the same mangled-cwd lookup. Copies held by both devices are folded by basename (larger file wins, size tie → lexicographically smaller path), so a pattern count never sees one conversation twice. Unset → live stores only; the variable is the switch and `~/.env.local` is its SSOT, so that one key is read out of the file when the variable is *absent* from the environment (a login-captured env predating the line would otherwise disable the corpus silently — measured 2026-09-03). A device records where a session was **collected**, not where it was created — do not weight by it. ## How It Works Each session is a JSONL file capturing tool calls, tool results (with success/failure), user messages, assistant prose, and compaction summaries. Patterns across sessions show where the agent repeatedly struggles. The two harnesses record the same events under different names. What the adapter normalizes — worth knowing when you drop to raw JSONL in Step 3b: | Signal | pi | Claude Code | |---|---|---| | tool call | `toolCall`, `arguments.path` | `tool_use`, `input.file_path` | | tool result | `role: toolResult`, `isError` | `tool_result` block in a **user** message, `is_error` | | user abort | `stopReason: "aborted"` | text `[Request interrupted by user]` | | **permission denial** | — | `is_error` result: *"user doesn't want to proceed"* | | compaction | `type: "compaction"` | user record, `isCompactSummary: true` | Claude Code carries one signal pi does not: a **denied permission prompt** — the user reading a proposed tool call and pressing No. It is reported as a correction, not a failure: nothing broke, the agent was about to do the wrong thing. High-value, and easy to lose in the failure stats if you don't split it. ## Extraction Script ```bash python3 {baseDir}/extract.py [options] ``` Auto-discovers the sessions directory from `$PWD` for the current harness. Use `--source` to pick a harness, `--sessions-dir` to point somewhere explicit. Changing `extract.py`? Run the regression suite — it pins the clock, the prose/thinking split, and the denial-vs-failure boundary, all of which have broken before: ```bash python3 {baseDir}/test_extract.py ``` ### Modes | Mode | What it extracts | |------|------------------| | `--summary` | Overview: session count, tool usage, failure count, abort count | | `--commands --stats` | Most common bash commands (frequency table) | | `--reads --stats` | Most read files | | `--says --match REGEX` | What the agent *said* — its prose. The only window on tone | | `--failures --stats` | Tool failures (`isError=true`) with triggering command context | | `--corrections` | User corrections: aborted agent turns paired with next user message | | `--sequences` | Narrative view: tool calls, user messages, failures in order | | `--sequences --match ERROR` | Zoom into error sequences with surrounding context | | `--compactions` | Session summaries: goals, progress, blockers, decisions | | `--context LINE` | Full untruncated context around a specific line in a session file | ### Common Options | Flag | Description | |------|-------------| | `--source pi\|claude\|all` | Harness to analyze (default: the one you're running under) | | `--match REGEX` | Filter items by regex | | `--stats` | Frequency table instead of raw output | | `--last N` | Number of recent sessions (default: 10) | | `--top N` | Items in frequency table (default: 30) | | `--before DATE` | Only sessions before this date (ISO: 2026-03-01) | | `--after DATE` | Only sessions on or after this date (ISO: 2026-03-01) | | `--include-heuristic` | With `--failures`: also show pattern-matched output (noisy) | | `--sessions-dir PATH` | Override auto-discovered sessions dir | | `--projects DIR [DIR ...]` | Analyze sessions from multiple project directories | | `--session-file PATH` | Session file path (required with `--context`) | | `--window N` | Entries before/after `--context` line (default: 5) | ### Output Format All output includes JSONL line references (`L:NNN` or `session:LNNN`) and the **full filepath** to the session file (as a header per session, or as a legend in stats mode). This lets you jump from any finding directly to the raw data. To drill into a specific event with the built-in context viewer: ```bash python3 {baseDir}/extract.py --context 42 --session-file /path/to/session.jsonl ``` Or manually with jq/sed: ```bash sed -n '42p' /path/to/session.jsonl | python3 -m json.tool ``` ## Workflow Follow these steps in order. Present findings to the user after each step. ### Step 1: Overview and Context ```bash python3 {baseDir}/extract.py --summary ``` Read the project's `AGENTS.md` if it exists. Understand what guidance the agent already has. ### Step 2: Find Recurring Patterns Run the frequency analyses and check user corrections: ```bash python3 {baseDir}/extract.py --commands --stats python3 {baseDir}/extract.py --failures --stats python3 {baseDir}/extract.py --reads --stats python3 {baseDir}/extract.py --corrections ``` Look for: - **High frequency, many sessions**: agent doing the same thing over and over - **Recurring failures**: same errors across sessions - **Repeated file reads**: agent can't find what it needs - **Command variations**: same intent, many spellings (e.g. `make test | tail -5`, `make test | tail -10`, `make test | tail -20` — noisy output problem) - **User corrections**: what the user aborted and redirected — these reveal cases where the agent technically succeeded but did the wrong thing ### Step 3: Understand the Stories For the top patterns, use sequences to see *what happened*: ```bash # See error narratives python3 {baseDir}/extract.py --sequences --match "ERROR" # Deep-dive into specific patterns python3 {baseDir}/extract.py --commands --match "git add" python3 {baseDir}/extract.py --failures --match "syntax|paren|not found" ``` The sequence view shows: - `USER` messages — what the user asked for or complained about - `BASH/EDIT/READ/WRITE` — what the agent did - `!! ERROR` — where things went wrong (ground truth: non-zero exit / tool error) - Context before and after failures reveals the root cause Also check compaction summaries for session-level context: ```bash python3 {baseDir}/extract.py --compactions ``` ### Step 3a: Zoom Into Specific Moments When a sweep surfaces something interesting at a specific line, use `--context` to see the full untruncated picture — complete tool output, full user messages, full assistant reasoning and thinking: ```bash # The filepath is shown in every session header — copy it directly python3 {baseDir}/extract.py --context 42 --session-file /path/to/session.jsonl # Wider window for complex sequences python3 {baseDir}/extract.py --context 42 --session-file /path/to/session.jsonl --window 10 ``` This is the primary drill-down tool. Use it whenever a line number catches your attention in the sweep output. ### Step 3b: Go Off-Script — Investigate the Raw JSONL `--context` covers most drill-down needs, but sometimes you need to ask questions it can't answer — correlating events far apart in a session, counting patterns across the whole file, or extracting specific fields. For those, go straight to the JSONL with jq, grep, or python one-liners. **Mind the schema.** The recipes below are **pi-shaped**. Run them against a Claude Code file and they return nothing — which reads like "no problems found" and is the easiest way to draw a false conclusion here. `extract.py` hides this difference; raw `jq` does not. Check which harness the file belongs to first — the `.pi/agent/sessions/` vs `.claude/projects/` segment, which holds for corpus paths too (`<corpus>/<device>/.claude/projects/…`) — and use the matching column: | | pi (`~/.pi/agent/sessions/<mangled>/`) | Claude Code (`~/.claude/projects/<mangled>/`) | |---|---|---| | record | `.type == "message"` | `.type == "user"` / `"assistant"` | | role | `.message.role` (incl. `"toolResult"`) | `.message.role` (no toolResult role) | | tool call | `.type == "toolCall"`, `.arguments` | `.type == "tool_use"`, `.input` | | tool result | role `toolResult`, `.message.isError` | `.type == "tool_result"` block **inside a user message**, `.is_error` | | tool name on a result | `.message.toolName` | absent — join `.tool_use_id` → the `tool_use` `.id` | | abort | `.message.stopReason == "aborted"` | text `[Request interrupted by user]` | Claude Code file paths also appear under per-session UUID subdirs; `subagents/` holds Task sidechains (a different agent's story — exclude unless that's the target). Example investigations (pi schema): ```bash # Get full context around a suspicious line S=~/.pi/agent/sessions/<dir>/<file>.jsonl sed -n '40,50p' "$S" | jq -r '.message.content[]?.text // empty' | head -40 # All user messages (complaints, corrections, instructions) jq -r 'select(.type=="message") | select(.message.role=="user") | .message.content[]? | select(.type=="text") | .text' "$S" # Full error output for a specific toolResult (not truncated) sed -n '42p' "$S" | jq -r '.message.content[].text' # All tool calls in order with their names (quick narrative) jq -r 'select(.type=="message") | select(.message.role=="assistant") | .message.content[]? | select(.type=="toolCall") | "\(.name): \(.arguments | tostring | .[0:120])"' "$S" # Count consecutive edits to the same file (struggle detector) jq -r 'select(.type=="message") | select(.message.role=="assistant") | .message.content[]? | select(.type=="toolCall") | select(.name=="edit") | .arguments.path' "$S" \ | uniq -c | sort -rn | head # All toolResult errors with full output jq -r 'select(.type=="message") | select(.message.role=="toolResult") | select(.message.isError==true) | "[\(.message.toolName)] \(.message.content[0].text[0:300])"' "$S" # What did the assistant say right after an error? (reaction pattern) # Use line numbers: if error is at L42, check L43 sed -n '43p' "$S" | jq -r '.message.content[]? | select(.type=="text") | .text[0:300]' # Find retry/struggle loops: same command repeated within 10 lines jq -r 'select(.type=="message") | select(.message.role=="assistant") | .message.content[]? | select(.type=="toolCall") | select(.name=="bash") | .arguments.command' "$S" \ | uniq -c | sort -rn | head ``` The same investigations against a **Claude Code** session: ```bash S=~/.claude/projects/<dir>/<uuid>.jsonl # All tool calls in order (quick narrative) jq -r 'select(.type=="assistant") | .message.content[]? | select(.type=="tool_use") | "\(.name): \(.input | tostring | .[0:120])"' "$S" # Failed tool results, full output. Add `| select(test("doesn.t want to # proceed"))` to isolate permission denials — the user pressed No. jq -r 'select(.type=="user") | .message.content[]? | select(.type=="tool_result" and .is_error==true) | (.content | if type=="array" then .[0].text else . end)' "$S" # What the user actually says (drop injected context and tool results) jq -r 'select(.type=="user" and (.isMeta|not)) | .message.content | if type=="string" then . else (.[]? | select(.type=="text") | .text) end' "$S" ``` Note `isMeta` records: local-command caveats and skill preambles injected into the transcript. They are *not* the user talking, and they outnumber real user messages — filter them out or your "what did the user complain about" query drowns. Trust your judgment. If extract.py's output raises a question, answer it from the data — the JSONL has full tool output, user messages, and assistant reasoning.