improve-agent

name

improve-agent

description

Analyze past session files (pi or Claude Code) to find recurring AI agent issues and fix them via AGENTS.md updates, new skills, or code/infra changes. Use when asked to improve agent workflow, find recurring problems, optimize AGENTS.md, create skills from session patterns, or understand what went wrong across sessions. Also covers tone — when the complaint is that the agent *sounded* defeated, self-critical, or that collaboration felt heavy, use `--says` and the opening-frame method in Step 3c rather than word counts.

Improve Agent

Analyze past coding sessions to find recurring agent issues, then fix them by updating AGENTS.md, creating new skills, or improving code/infra.

Multi-harness. Both pi (~/.pi/agent/sessions/<mangled-cwd>/) and Claude Code (~/.claude/projects/<mangled-cwd>/) are supported. extract.py translates Claude Code records into the pi schema on read, so every mode below works on either source. Default source is the harness you are running under; override with --source.

Multi-device. With ANDENKEN_SESSION_CORPUS set (~/.env.local, the same variable andenken's indexer and session-recap read), the gathered corpus is scanned in addition to the live stores — the corpus keeps each runtime's path shape under <corpus>/<device>/, so the other machine's sessions for the same project are found by the same mangled-cwd lookup. Copies held by both devices are folded by basename (larger file wins, size tie → lexicographically smaller path), so a pattern count never sees one conversation twice. Unset → live stores only; the variable is the switch and ~/.env.local is its SSOT, so that one key is read out of the file when the variable is absent from the environment (a login-captured env predating the line would otherwise disable the corpus silently — measured 2026-09-03). A device records where a session was collected, not where it was created — do not weight by it.

How It Works

Each session is a JSONL file capturing tool calls, tool results (with success/failure), user messages, assistant prose, and compaction summaries. Patterns across sessions show where the agent repeatedly struggles.

The two harnesses record the same events under different names. What the adapter normalizes — worth knowing when you drop to raw JSONL in Step 3b:

Signal	pi	Claude Code
tool call	toolCall, arguments.path	tool_use, input.file_path
tool result	role: toolResult, isError	tool_result block in a user message, is_error
user abort	stopReason: "aborted"	text [Request interrupted by user]
permission denial	—	is_error result: "user doesn't want to proceed"
compaction	type: "compaction"	user record, isCompactSummary: true

Claude Code carries one signal pi does not: a denied permission prompt — the user reading a proposed tool call and pressing No. It is reported as a correction, not a failure: nothing broke, the agent was about to do the wrong thing. High-value, and easy to lose in the failure stats if you don't split it.

Extraction Script

python3 {baseDir}/extract.py [options]

Auto-discovers the sessions directory from $PWD for the current harness. Use --source to pick a harness, --sessions-dir to point somewhere explicit.

Changing extract.py? Run the regression suite — it pins the clock, the prose/thinking split, and the denial-vs-failure boundary, all of which have broken before:

python3 {baseDir}/test_extract.py

Modes

Mode	What it extracts
--summary	Overview: session count, tool usage, failure count, abort count
--commands --stats	Most common bash commands (frequency table)
--reads --stats	Most read files
--says --match REGEX	What the agent said — its prose. The only window on tone
--failures --stats	Tool failures (isError=true) with triggering command context
--corrections	User corrections: aborted agent turns paired with next user message
--sequences	Narrative view: tool calls, user messages, failures in order
--sequences --match ERROR	Zoom into error sequences with surrounding context
--compactions	Session summaries: goals, progress, blockers, decisions
--context LINE	Full untruncated context around a specific line in a session file

Common Options

Flag	Description
--source pi|claude|all	Harness to analyze (default: the one you're running under)
--match REGEX	Filter items by regex
--stats	Frequency table instead of raw output
--last N	Number of recent sessions (default: 10)
--top N	Items in frequency table (default: 30)
--before DATE	Only sessions before this date (ISO: 2026-03-01)
--after DATE	Only sessions on or after this date (ISO: 2026-03-01)
--include-heuristic	With --failures: also show pattern-matched output (noisy)
--sessions-dir PATH	Override auto-discovered sessions dir
--projects DIR [DIR ...]	Analyze sessions from multiple project directories
--session-file PATH	Session file path (required with --context)
--window N	Entries before/after --context line (default: 5)

Output Format

All output includes JSONL line references (L:NNN or session:LNNN) and the full filepath to the session file (as a header per session, or as a legend in stats mode). This lets you jump from any finding directly to the raw data.

To drill into a specific event with the built-in context viewer:

python3 {baseDir}/extract.py --context 42 --session-file /path/to/session.jsonl

Or manually with jq/sed:

sed -n '42p' /path/to/session.jsonl | python3 -m json.tool

Workflow

Follow these steps in order. Present findings to the user after each step.

Step 1: Overview and Context

python3 {baseDir}/extract.py --summary

Read the project's AGENTS.md if it exists. Understand what guidance the agent already has.

Step 2: Find Recurring Patterns

Run the frequency analyses and check user corrections:

python3 {baseDir}/extract.py --commands --stats
python3 {baseDir}/extract.py --failures --stats
python3 {baseDir}/extract.py --reads --stats
python3 {baseDir}/extract.py --corrections

Look for:

• High frequency, many sessions: agent doing the same thing over and over
• Recurring failures: same errors across sessions
• Repeated file reads: agent can't find what it needs
• Command variations: same intent, many spellings (e.g. make test | tail -5, make test | tail -10, make test | tail -20 — noisy output problem)
• User corrections: what the user aborted and redirected — these reveal cases where the agent technically succeeded but did the wrong thing

Step 3: Understand the Stories

For the top patterns, use sequences to see what happened:

# See error narratives
python3 {baseDir}/extract.py --sequences --match "ERROR"

# Deep-dive into specific patterns
python3 {baseDir}/extract.py --commands --match "git add"
python3 {baseDir}/extract.py --failures --match "syntax|paren|not found"

The sequence view shows:

• USER messages — what the user asked for or complained about
• BASH/EDIT/READ/WRITE — what the agent did
• !! ERROR — where things went wrong (ground truth: non-zero exit / tool error)
• Context before and after failures reveals the root cause

Also check compaction summaries for session-level context:

python3 {baseDir}/extract.py --compactions

Step 3a: Zoom Into Specific Moments

When a sweep surfaces something interesting at a specific line, use --context to see the full untruncated picture — complete tool output, full user messages, full assistant reasoning and thinking:

# The filepath is shown in every session header — copy it directly
python3 {baseDir}/extract.py --context 42 --session-file /path/to/session.jsonl

# Wider window for complex sequences
python3 {baseDir}/extract.py --context 42 --session-file /path/to/session.jsonl --window 10

This is the primary drill-down tool. Use it whenever a line number catches your attention in the sweep output.

Step 3b: Go Off-Script — Investigate the Raw JSONL

--context covers most drill-down needs, but sometimes you need to ask questions it can't answer — correlating events far apart in a session, counting patterns across the whole file, or extracting specific fields. For those, go straight to the JSONL with jq, grep, or python one-liners.

Mind the schema. The recipes below are pi-shaped. Run them against a Claude Code file and they return nothing — which reads like "no problems found" and is the easiest way to draw a false conclusion here. extract.py hides this difference; raw jq does not. Check which harness the file belongs to first — the .pi/agent/sessions/ vs .claude/projects/ segment, which holds for corpus paths too (<corpus>/<device>/.claude/projects/…) — and use the matching column:

	pi (~/.pi/agent/sessions/<mangled>/)	Claude Code (~/.claude/projects/<mangled>/)
record	.type == "message"	.type == "user" / "assistant"
role	.message.role (incl. "toolResult")	.message.role (no toolResult role)
tool call	.type == "toolCall", .arguments	.type == "tool_use", .input
tool result	role toolResult, .message.isError	.type == "tool_result" block inside a user message, .is_error
tool name on a result	.message.toolName	absent — join .tool_use_id → the tool_use .id
abort	.message.stopReason == "aborted"	text [Request interrupted by user]

Claude Code file paths also appear under per-session UUID subdirs; subagents/ holds Task sidechains (a different agent's story — exclude unless that's the target).

Example investigations (pi schema):

# Get full context around a suspicious line
S=~/.pi/agent/sessions/<dir>/<file>.jsonl
sed -n '40,50p' "$S" | jq -r '.message.content[]?.text // empty' | head -40

# All user messages (complaints, corrections, instructions)
jq -r 'select(.type=="message") | select(.message.role=="user")
  | .message.content[]? | select(.type=="text") | .text' "$S"

# Full error output for a specific toolResult (not truncated)
sed -n '42p' "$S" | jq -r '.message.content[].text'

# All tool calls in order with their names (quick narrative)
jq -r 'select(.type=="message") | select(.message.role=="assistant")
  | .message.content[]? | select(.type=="toolCall")
  | "\(.name): \(.arguments | tostring | .[0:120])"' "$S"

# Count consecutive edits to the same file (struggle detector)
jq -r 'select(.type=="message") | select(.message.role=="assistant")
  | .message.content[]? | select(.type=="toolCall")
  | select(.name=="edit") | .arguments.path' "$S" \
  | uniq -c | sort -rn | head

# All toolResult errors with full output
jq -r 'select(.type=="message") | select(.message.role=="toolResult")
  | select(.message.isError==true)
  | "[\(.message.toolName)] \(.message.content[0].text[0:300])"' "$S"

# What did the assistant say right after an error? (reaction pattern)
# Use line numbers: if error is at L42, check L43
sed -n '43p' "$S" | jq -r '.message.content[]?
  | select(.type=="text") | .text[0:300]'

# Find retry/struggle loops: same command repeated within 10 lines
jq -r 'select(.type=="message") | select(.message.role=="assistant")
  | .message.content[]? | select(.type=="toolCall")
  | select(.name=="bash") | .arguments.command' "$S" \
  | uniq -c | sort -rn | head

The same investigations against a Claude Code session:

S=~/.claude/projects/<dir>/<uuid>.jsonl

# All tool calls in order (quick narrative)
jq -r 'select(.type=="assistant") | .message.content[]? | select(.type=="tool_use")
  | "\(.name): \(.input | tostring | .[0:120])"' "$S"

# Failed tool results, full output. Add `| select(test("doesn.t want to
# proceed"))` to isolate permission denials — the user pressed No.
jq -r 'select(.type=="user") | .message.content[]?
  | select(.type=="tool_result" and .is_error==true)
  | (.content | if type=="array" then .[0].text else . end)' "$S"

# What the user actually says (drop injected context and tool results)
jq -r 'select(.type=="user" and (.isMeta|not)) | .message.content
  | if type=="string" then . else (.[]? | select(.type=="text") | .text) end' "$S"

Note isMeta records: local-command caveats and skill preambles injected into the transcript. They are not the user talking, and they outnumber real user messages — filter them out or your "what did the user complain about" query drowns.

Trust your judgment. If extract.py's output raises a question, answer it from the data — the JSONL has full tool output, user messages, and assistant reasoning.