| name | cog-tune |
| description | Analyze ClosePaw agent cognition using debug-run traces/replay artifacts and eval results, then propose and implement improvements to prompts, tool definitions, context packing (todo/scratchpad/history), and multi-agent coordination. Use when a debug run feels wrong, when eval metrics regress, when tuning context engineering for generalizable gains, or when reviewing LLM input/output and tool usage; produce both a report and code/doc changes. |
Cog Tune
Overview
Improve agent cognition by inspecting debug-run traces, eval metrics/results, screenshots/screen observations, LLM inputs/outputs, and tool calls, then applying generalizable prompt/context/tool changes backed by evidence.
Quick Debug (lightweight entry)
For simple "agent did something wrong" cases that don't need full eval analysis, use this abbreviated flow:
- Run debug session:
./scripts/setup.sh && ./scripts/debug-run.sh "goal"
- Turn-by-turn check: For each turn, inspect all three layers — even if the tool reports success:
- Reasoning: Does the agent choose the right action given the screen state?
- Grounding: Are the action type and parameters correct? (right element, right text, right coordinates)
- Execution: Compare
turn_N.png (before) vs turn_N+1.png (after) — did the screen actually change as expected? Do NOT trust tool success status alone; inspect screenshots when in doubt.
- Observation: Does the agent correctly interpret the post-action state?
- Quick diagnostics:
grep -E "click|type|scroll|swipe|back|home" debug-output/run_*/agent.log
grep "ActionResult\|ToolCallResult" debug-output/run_*/agent.log
grep "ERROR\|Exception" debug-output/run_*/agent.log
- Fix and verify: Apply targeted fix, then re-run
./scripts/setup.sh && ./scripts/debug-run.sh "goal".
If the issue is unclear after this quick pass, proceed to the full workflow below.
Full Workflow
1. Set scope and guardrails
- Clarify the failing behavior, target runs, and target metric(s).
- If tuning from eval, define baseline run and comparison run up front.
- Prefer generalizable changes; avoid one-off hacks for a single task.
- Require evidence from traces and/or eval artifacts: show which step(s), run IDs, and metrics support each proposed change.
2. Prepare evidence data
Virtualenvs — two separate venvs, use the right one:
eval/.venv/bin/python — for all scripts under eval/inspection_tool/.venv/bin/python — for all scripts under inspection_tool/
Pick one or both entry points:
3. Inspect cognition step-by-step
Use trace/derived/steps.jsonl plus artifacts in trace/artifacts/:
- World:
screenshot, tool_observation_screen, raw_a11y_tree, sanitized_a11y_tree
- Mind:
llm_system_prompt, llm_user_context, llm_full_prompt, llm_input_items, llm_history, llm_tool_calls
- Act/Observe:
tool_call_args, tool_result, tool_observation_screen
CRITICAL — Verify every turn, not just failed ones:
For each turn, evaluate all three layers:
- Reasoning: Did the agent choose the correct action given the screen state and goal? Was the logic sound?
- Grounding: Did the agent select the right action type and parameters to implement its reasoning? (e.g., correct element ID, correct text, correct coordinates)
- Execution: Did the action actually succeed on the device? Do NOT trust
tool_result alone — a tool may report success while the UI did not change. Always compare the before screenshot (turn_N.png) with the after screenshot (turn_N+1.png or tool_observation_screen). If in doubt, inspect the screenshot image directly to confirm the screen state changed as expected.
Always cross-check image evidence with a11y trees. If they disagree, document the mismatch explicitly and avoid conclusions based on only one modality.
Look for mismatches between:
- Screen state vs. what the model believed
- Tool call args vs. available UI elements
- History/todo/scratchpad vs. chosen action
- Planner vs. executor handoff (delegation summaries)
- Eval-level regressions vs. per-task cognition patterns (for example: lower success rate tied to repeated tool mis-targeting)
4. Classify root cause
Bucket issues before changing prompts:
- Perception (missing/incorrect a11y data)
- Context (missing/overloaded system/user context)
- Reasoning (bad choice despite correct inputs)
- Execution (tool call failure or wrong target) — use
./scripts/action-test.sh to isolate and reproduce action-level failures independently of the agent
- Observation (post-action state not captured)
- Orchestration (multi-agent handoff gaps)
- Evaluation gap (metric selection/run config mismatch, flaky task set, or benchmark harness artifact) — use
eval/aw_bridge/setup_task_only.py --task <TaskName> to run task setup in isolation and manually verify the environment state
5. Apply changes (minimal, generalizable)
Possible change areas:
- Prompt assembly:
agent/cognition/prompt/PromptAssembler.kt, AgentPromptBuilder.kt
- Context packing:
agent/cognition/context/ContextPackager.kt
- Policies:
agent/cognition/policy/*
- Tool schemas:
tool/ToolSpec.kt, tool impls under tool/impl/
- Delegation:
tool/impl/DelegateTaskTool.kt, agent/subagent/*
Use rg to locate prompt or tool definition text before edits.
6. Validate and generalize
Output format
Per-task analysis: Follow the template in assets/per_task_analysis_template.md.
Common problems summary (when synthesizing for /autotune): Follow .claude/skills/autotune/assets/common_problems_template.md.
Overall report (when doing a full analysis, not per-task):
- Summary: what went wrong, and evidence (step IDs, artifact paths, run IDs, metric deltas)
- Root cause: category + reasoning
- Proposed changes: concise list with impacted files
- Patch: actual code/doc updates
- Verification plan: debug runs and/or eval runs to re-test
Project references
- Debug workflow guide:
doc/dev/visual_debug_guide.md
- Debug run script:
scripts/debug-run.sh
- Action test harness:
scripts/action-test.sh
- Replay compiler:
inspection_tool/replay_compiler.py
- Token stats analyzer:
inspection_tool/a11y_token_stats.py
- Eval harness:
eval/README.md
- Eval architecture:
doc/main/eval/eval.md
- Eval runner:
eval/aw_bridge/runner.py
- Eval remote config:
eval/config/remote.yaml
- Remote eval: see
/autotune skill Step 3
- Eval bridge config:
eval/aw_bridge/native_agent_bridge.py (agent_mode, perception_mode, platform_mode, excluded_tools, model selection)
- Eval completion monitor:
eval/aw_bridge/completion_monitor.py
- Eval preflight / snapshot policy:
eval/aw_bridge/runner_preflight.py
- Eval per-task lifecycle:
eval/aw_bridge/runner_execution.py
- Eval task setup (standalone):
eval/aw_bridge/setup_task_only.py
- Eval summarizer:
eval/analysis/summarize.py
- Eval run comparator:
eval/analysis/compare_runs.py
Related skills
/prompt-tune — Apply prompt, tool description, and app skill changes based on this skill's diagnosis. Enforces the three-layer ownership model. Use after analysis is complete.
