| name | dev-eval |
| description | Run comparative development evaluations: assign the same task to agent-worker team AND
claude-code (team subagents), then have an evaluator agent score both and a reviewer agent
verify the evaluation. Fully orchestrated — you coordinate all phases, agents notify via
channel/callback when done. Use this skill when the user wants to benchmark agent-worker
vs claude-code, run a dev eval, compare agent performance, or prove that the workspace
team outperforms solo agents. Trigger on phrases like 'dev eval', 'run eval', 'compare
performance', 'benchmark', '对比评估', '开发评估', '跑评估', 'eval iteration', '/dev-eval'.
|
CLI path: All bash code blocks use $AW_CMD for the agent-worker CLI.
Define it at the start of any shell session before running commands:
AW_CMD="/Users/lidessen/.bun/bin/bun /Users/lidessen/workspaces/agent-worker/packages/agent-worker/src/cli/index.ts"
Statistical significance notice: A single eval run is anecdotal. Conclusions require
≥3 independent runs of the same task config before attributing score differences to
architecture rather than noise. Track runs in SUMMARY.md and note N before drawing
conclusions.
Dev Eval: Comparative Development Evaluation
Run the same development task through two execution paths in parallel, then have
agents evaluate and review the results.
┌─────────────────────┐
│ Task Specification │
└──────────┬──────────┘
│
┌────────────────┴────────────────┐
▼ ▼
┌──────────────────────┐ ┌──────────────────────────┐
│ Agent-Worker Team │ │ Claude-Code Team Agents │
│ $AW_CMD run eval.yaml │ │ (Plan → Impl → Review) │
│ (both in worktrees) │ │ isolation: "worktree" │
└──────────┬───────────┘ └─────────────┬────────────┘
│ ← process exits │ ← agent returns
│ │
└──────────────┬──────────────────┘
▼
┌─────────────────────────┐
│ Evaluator Agent │
│ (scores both sides) │
└────────────┬────────────┘
▼
┌─────────────────────────┐
│ Review Agent │
│ (validates scoring) │
└────────────┬────────────┘
▼
┌─────────────────────────┐
│ Final Report │
└─────────────────────────┘
Phase 0: Task Definition
Write a task spec as a file. Both sides get identical instructions via file reference.
# Eval Task: [slug]
## Task
[one-line summary]
## Requirements
- [requirement 1]
- [requirement 2]
## Acceptance Criteria
- [criterion 1]
- [criterion 2]
## Tier
T1 / T2 / T3
Save to .agent-workspace/evals/YYYY-MM-DD-{slug}/task-spec.md.
IMPORTANT: Always save the task spec as a file first. The workspace YAML reads it
via a setup step (cat task-spec.md) to avoid shell escaping issues with --var.
Blind-Test Mechanism
Before passing artifacts to the evaluator, randomly assign neutral labels
to prevent anchoring bias:
if [ $(($RANDOM % 2)) -eq 0 ]; then
LABEL_TEAM="Solution A"; LABEL_SOLO="Solution B"
else
LABEL_TEAM="Solution B"; LABEL_SOLO="Solution A"
fi
echo "LABEL_TEAM=$LABEL_TEAM" > $EVAL_DIR/blind-mapping.txt
echo "LABEL_SOLO=$LABEL_SOLO" >> $EVAL_DIR/blind-mapping.txt
Pass only $LABEL_TEAM / $LABEL_SOLO labels to the evaluator — never
mention "agent-worker", "team", or "solo". After the evaluator returns scores,
reveal the mapping and rewrite the scorecard with real names.
Baseline Isolation
Both sides run in git worktrees to ensure clean, auditable comparisons.
Before launching either run, record the baseline:
EVAL_DIR=.agent-workspace/evals/YYYY-MM-DD-slug
BASELINE_SHA=$(git rev-parse HEAD)
echo "$BASELINE_SHA" > $EVAL_DIR/baseline-sha.txt
git status --porcelain > $EVAL_DIR/baseline-status.txt
- Agent-Worker Team:
aw run operates in the main repo by default, but we create
a worktree for it (see Path A below) so its diff is clean.
- Claude-Code Team: Agent tool with
isolation: "worktree" — already isolated.
Both diffs are computed as git diff $BASELINE_SHA inside their respective worktrees.
This guarantees the patches contain only changes from that specific run.
Known Limitations
aw run task mode does not auto-complete. Workspace agent loops never self-stop —
they poll inbox indefinitely. checkCompletion() checks loop.isRunning which is
always true for active orchestrators. As a result, aw run --wait always times out.
Use service mode (aw create) + manual completion detection until this is fixed.
Kickoff message length limit. Channel send guard rejects messages > 1200 chars.
Do NOT inline task content in kickoff via ${{ task }}. Instead, reference the file
path and let agents read it themselves.
Solo diff baseline. The Agent tool's isolation: "worktree" creates a worktree
from the current branch. Use git diff HEAD inside the solo worktree (not git diff $BASELINE_SHA) to capture only the solo agent's changes.
deepseek runtime: Use deepseek:deepseek-chat as the model string with runtime: ai-sdk. Ensure DEEPSEEK_API_KEY is set in the environment.
T1 tasks: skip team structure. T1 tasks (add CLI command, fix bug, small config)
are too simple to benefit from team coordination. The orchestration overhead consistently
costs 1-2 points vs solo. Run T1 tasks as solo-only evals or skip Run A entirely.
kimi-code model string: Use kimi-code:kimi-for-coding (NOT moonshot:moonshot-v1-auto).
The moonshot provider is not registered — it will fail with "Unknown provider: moonshot".
Pre-check runtimes before starting an eval. Run aw ls to confirm all configured
agents are idle. Test each runtime independently if in doubt.
Phase 1: Parallel Execution
Launch both runs simultaneously. You (the orchestrator) manage both.
Path A: Agent-Worker Team — two modes
Option 1: aw run (task mode — CURRENTLY BROKEN)
Generate a workspace YAML config. The process blocks until all agents finish.
CRITICAL: aw run auto-removes the workspace after completion (mode: "task").
You must configure data_dir to persist logs, AND start a parallel log capture process
BEFORE running, so artifacts survive workspace removal.
Workspace isolation: Create a git worktree for the team run so its diff is clean:
EVAL_DIR=.agent-workspace/evals/YYYY-MM-DD-slug
TEAM_WORKTREE=/tmp/eval-team-$(date +%s)
git worktree add --detach $TEAM_WORKTREE $BASELINE_SHA
echo "$TEAM_WORKTREE" > $EVAL_DIR/team-worktree-path.txt
name: eval-team
data_dir: .agent-workspace/evals/YYYY-MM-DD-slug/workspace-data
agents:
claude-code:
runtime: claude-code
model: sonnet
instructions: |
You are the orchestrator (宰相). Break down the task, assign subtasks to
the other agents, review their outputs, and coordinate to completion.
Review is only valuable when it closes the loop. Finding a bug and then
stopping is equivalent to not reviewing at all — the codebase ends up in
the same broken state. A complete review cycle is:
review → find issue → delegate fix to implementer → implementer fixes →
you verify the fix → only then declare done.
If you exit after finding issues without fixing them, the team's effort
produces no better outcome than working alone.
IMPORTANT — Completion protocol:
1. Do NOT stop until ALL subtasks are implemented AND reviewed.
2. If review finds issues, assign fix tasks back to the implementer and
re-review after fixes land. Repeat until all dimensions pass.
3. After ALL dimensions pass, post EVAL_COMPLETE_TEAM to general channel.
If blocked and cannot complete, post EVAL_COMPLETE_TEAM status=blocked
reason=<why> so the run can be scored.
4. You must be the LAST agent to stop. Verify all others are done first.
kimi-code:
runtime: ai-sdk
model: kimi-code:kimi-for-coding
instructions: |
You are an executor. Complete the subtasks assigned by @claude-code.
Report back when done. Do NOT stop until @claude-code confirms your work.
Before implementing, confirm TypeScript types by reading the relevant
source files. Do NOT use `as any` casts — find the correct type first.
channels:
- general
- code-review
setup:
- shell: cat ${{ eval_dir }}/task-spec.md
as: task
kickoff: |
@claude-code 以下是开发任务，请分解并协调团队完成：
${{ task }}
项目路径: ${{ project_path }}
Then run with parallel log capture:
EVAL_DIR=.agent-workspace/evals/YYYY-MM-DD-slug
$AW_CMD log @eval-team -f --json > $EVAL_DIR/team-log.jsonl
$AW_CMD run $EVAL_DIR/eval-workspace.yaml \
--var eval_dir="$EVAL_DIR" \
--var project_path="$TEAM_WORKTREE" \
--wait 30m
Log capture timing: aw log @eval-team -f opens an SSE stream that waits for the
workspace to appear if it doesn't exist yet. The daemon creates the workspace on
aw run, and the log stream attaches as soon as the workspace key becomes available.
Start the log process first to avoid missing early events. The -f flag is required
— without it, aw log does a one-shot read of existing events and exits immediately,
which will produce an empty or incomplete log file.
Why parallel log capture? aw run with mode: "task" auto-removes the workspace
on completion. After removal, aw log and MCP tools can no longer access workspace
data. The data_dir config persists raw data files, and the parallel aw log -f --json
pipe captures the streaming event log before the workspace disappears.
Completion: aw run exits with:
0 — all agents stopped naturally (completed)1 — a loop errored (failed)2 — --wait timeout exceeded
Completion safety: checkCompletion() triggers when ALL agent loops stop running.
The orchestrator prompt explicitly requires @claude-code to be the last to stop
and to verify all work is done first. This prevents premature completion where an
executor stops but hasn't received review feedback yet.
Use --tag to run the same config multiple times:
$AW_CMD run eval-workspace.yaml --tag iter-1 --var eval_dir="..." --wait 30m
$AW_CMD run eval-workspace.yaml --tag iter-2 --var eval_dir="..." --wait 30m
Option 2: Service mode (RECOMMENDED until aw run is fixed)
Create a workspace with aw create, send the task, and monitor for completion.
This mode does NOT auto-remove — you can collect artifacts at leisure.
Target syntax: agent@workspace[#channel] — see target.ts.
$AW_CMD ls
$AW_CMD send "claude-code@eval-team" "$(cat .agent-workspace/evals/.../task-spec.md)
完成后请在 general 频道发送: EVAL_COMPLETE_TEAM
项目路径: /path/to/worktree"
Completion detection (service mode needs explicit signals):
$AW_CMD read "@eval-team#general" 5
agents
wait_inbox
Path B: Claude-Code Team Agents (worktree)
Use the Agent tool with isolation: "worktree" to run a full team pipeline
(Planner → Implementer → Reviewer) in an isolated git worktree. This mirrors what
the /team skill does, but in isolation.
Agent(
subagent_type: "general-purpose",
isolation: "worktree",
prompt: """
You are executing a development task using the team pipeline approach.
Follow the Planner → Implementer → Reviewer flow.
## Task
[paste task-spec.md content here]
## Instructions
1. PLAN: Analyze the codebase. Produce an implementation plan with phases,
files to modify, and key changes.
2. IMPLEMENT: Execute each phase. Read files before modifying.
Follow existing patterns and conventions from CLAUDE.md.
3. REVIEW: After implementation, review your own changes for correctness,
completeness, and code quality. Fix any critical issues.
4. COMMIT: After completing implementation and tests, commit all changes:
git add -A && git commit -m "feat: <brief task description>"
5. REPORT: When done, output a structured completion report:
- Files changed (with paths)
- Tests added/modified
- Key design decisions
- Any issues encountered and how resolved
- Time taken per phase (estimate)
- Total tool calls made (count from your conversation)
You may spawn sub-agents (Plan, general-purpose, feature-dev:code-reviewer)
to parallelize. Use the same flow as the /team skill.
"""
)
The Agent tool returns when done — this is the natural completion signal.
Parallelism
Launch both paths in the same message using parallel tool calls:
# Step 0 (before both runs): record baseline + create team worktree + save task spec
git rev-parse HEAD > baseline-sha.txt
git worktree add --detach /tmp/eval-team-$(date +%s) $(cat baseline-sha.txt)
Write task-spec.md to the eval dir
# Step 1 (parallel):
# Bash (run_in_background): start log capture (MUST use -f for streaming)
# $AW_CMD log @eval-team -f --json > $EVAL_DIR/team-log.jsonl
# Bash (run_in_background): $AW_CMD run
# $AW_CMD run $EVAL_DIR/eval-workspace.yaml --wait 30m
# Agent (worktree): Claude-Code team pipeline
# Agent(isolation: "worktree", prompt: "...")
The aw run Bash command blocks in background until workspace completes.
The Agent tool blocks until the worktree agent finishes.
You get notified when each completes — no polling needed.
While both run, you can optionally monitor workspace progress:
$AW_CMD ls → workspace status
$AW_CMD read "@eval-team#general" 5 → recent channel messages
Phase 2: Collect Artifacts
After both complete, gather the evidence for evaluation.
Agent-Worker artifacts
EVAL_DIR=.agent-workspace/evals/YYYY-MM-DD-slug
TEAM_WORKTREE=$(cat $EVAL_DIR/team-worktree-path.txt)
BASELINE_SHA=$(cat $EVAL_DIR/baseline-sha.txt)
cat $EVAL_DIR/team-log.jsonl | bun .claude/skills/dev-eval/scripts/eval-metrics.ts \
> $EVAL_DIR/team-metrics.txt
cd $TEAM_WORKTREE && git diff $BASELINE_SHA > $EVAL_DIR/diff-team.patch
ls $EVAL_DIR/workspace-data/
git worktree remove --force $TEAM_WORKTREE
Note on aw run mode: By this point the workspace is already removed. All data
comes from: (1) the parallel log capture file, (2) the data_dir persistent storage,
and (3) the git diff from the team worktree.
Note on service mode: Workspace is still alive — you can also use MCP tools:
channel_read(channel: "general", limit: 200)
events(limit: 200)
agent_activity(agent: "claude-code", limit: 50)
Claude-Code artifacts
The worktree agent returns:
- Its completion report (inline in result)
- The worktree path + branch (from isolation)
SOLO_WORKTREE=[worktree-path from agent result]
cd $SOLO_WORKTREE && git diff HEAD > $EVAL_DIR/diff-solo.patch
git log --oneline --all > $EVAL_DIR/solo-commits.txt
Evidence symmetry
Both sides must provide comparable evidence for fair evaluation. The orchestrator
should normalize artifacts before passing them to the evaluator:
| Evidence type | Run A (Team) | Run B (Solo) | How to align |
|---|
| Code diff | diff-team.patch | diff-solo.patch | Both from worktrees against same baseline SHA ✅ |
| Process log | team-log.jsonl (event stream) | Agent tool conversation (not captured) | Extract: total tool calls, errors, phases, duration from both |
| Activity log | aw log events: messages, tool calls, status changes, errors per agent | Agent return value (self-reported) | Team has full observability; Solo is self-reported. Note asymmetry. |
| Completion report | Channel messages (general) | Agent return value | Both should list: files changed, tests, decisions, issues |
| Cost | Event log has token/usage data | Agent doesn't report tokens | Note as "unavailable" for Run B; don't score what you can't measure |
Rule: If a metric is unavailable for one side, mark it "N/A" in the scorecard
rather than guessing. The evaluator scores only what evidence supports.
Activity log analysis
The team's event log (team-log.jsonl) is a rich source of process-quality signals
beyond the final diff. When preparing evidence for the evaluator, extract:
Collaboration quality:
- Message flow: who @mentioned whom, response latency between agents
- Task handoffs: did the orchestrator decompose clearly? Did executors stay in scope?
- Review loops: how many review cycles? What got caught?
Autonomy signals:
- Error → recovery patterns: did agents self-recover or get stuck?
- Idle gaps: long periods with no events = agent likely stuck
- Repeated tool calls on same file = potential thrashing
Efficiency signals:
- Parallel activity: overlapping timestamps across agents = concurrent work
- Total events per agent: high count on one agent = unbalanced load
- Tool call success rate: failures / total = quality of approach
Use the metrics script to extract these automatically:
cat $EVAL_DIR/team-log.jsonl | bun .claude/skills/dev-eval/scripts/eval-metrics.ts
For the Solo side, the Agent tool return value is the only process evidence.
Require the solo agent to self-report (via the REPORT step in its prompt):
tool call count, phases completed, errors encountered, and time estimates.
This is less reliable than observed data but the best available.
Phase 3: Evaluator Agent
Spawn an Evaluator Agent that scores both runs. This agent receives all artifacts
and produces the scorecard. It does NOT see which side it's "supposed" to favor — the
evaluation must be honest.
Agent(
subagent_type: "general-purpose",
prompt: """
You are an impartial development evaluation judge. Score two parallel
development runs that completed the same task.
## Task Specification
[task-spec.md content]
## Run A
- Process summary: [extracted from team-log.jsonl: phases, tool calls, errors, duration]
- Activity log highlights: [key events — message flow between agents, task handoffs,
review cycles, error→recovery patterns, idle gaps, parallel activity timestamps]
- Completion report: [final summary from general channel]
- Code diff: [diff-team.patch content or summary]
- Human interventions: N
- Wall-clock time: Xm
## Run B
- Process summary: [extracted from agent conversation: phases, tool calls, errors, duration]
- Activity log: [self-reported — tool call count, errors, phase timing. NOTE: less
reliable than observed data; weight accordingly]
- Completion report: [agent's structured report]
- Code diff: [diff-solo.patch content or summary]
- Human interventions: N
- Wall-clock time: Xm
## Scoring Rubric (1-5 per dimension)
**Autonomy**: How independently did it complete? Self-recovery from errors?
- 1: Blocked entirely — required human to unblock every step
- 2: Stuck once — required one human nudge to continue
- 3: Minor nudges — asked clarifying questions but self-directed
- 4: Fully autonomous — completed without intervention
- 5: Self-recovered + proactive — caught and fixed own errors, anticipated edge cases
**Quality**: Correctness, tests, code review findings?
- 1: Broken — tests fail or core requirement unmet
- 2: Works but messy — functional but with bugs or missing tests
- 3: Functional — all requirements met, basic test coverage
- 4: Clean + tested — solid implementation, good test coverage, handles edge cases
- 5: Exceptional — production-ready, comprehensive tests, handles all edge cases
**Speed**: Wall-clock efficiency, parallelism utilized?
- 1: >5x slower than expected for the task complexity
- 2: 2-5x slower — significant sequential bottlenecks
- 3: Expected duration for the task tier
- 4: Faster than expected — good parallelism or efficient tool use
- 5: Much faster — optimal parallelism, minimal wasted steps
**Maintainability**: Code readability, naming, structure, test coverage for future changes?
- 1: Unreadable — magic values, no structure, zero tests
- 2: Partially readable — some naming issues, minimal tests
- 3: Readable — consistent style, basic test coverage
- 4: Clean — clear abstractions, good test coverage, easy to extend
- 5: Exemplary — self-documenting, comprehensive tests, designed for change
**Completeness**: Requirements coverage, edge cases?
- 1: Major gaps — core requirements unmet
- 2: Core only — primary requirements done, secondary ignored
- 3: All requirements — full spec coverage
- 4: + edge cases — spec coverage plus error handling and edge cases
- 5: + proactive improvements — spec plus meaningful enhancements not asked for
**Cost**: Token/resource efficiency relative to output quality?
- 1: >10x wasteful — enormous token spend for minimal output
- 2: 3-10x wasteful — clearly over-engineered approach
- 3: Reasonable — proportionate token use for output quality
- 4: Efficient — achieved equivalent quality with notably fewer tokens
- 5: Remarkably efficient — high output quality per token
- If cost data is unavailable for a side, mark N/A and exclude from total.
## Output Format
Produce EXACTLY this format:
### Scorecard
| Dimension | Run A | Run B | Delta | Justification |
|---------------|:---:|:---:|:---:|---|
| Autonomy | X/5 | X/5 | +/-N | [why, citing specific evidence] |
| Quality | X/5 | X/5 | +/-N | [why] |
| Speed | X/5 | X/5 | +/-N | [why] |
| Maintainability | X/5 | X/5 | +/-N | [why] |
| Completeness | X/5 | X/5 | +/-N | [why] |
| Cost | X/5 or N/A | X/5 or N/A | +/-N | [why, or "insufficient data"] |
| **TOTAL** | **/N** | **/N** | **+/-N** | (denominator excludes N/A dimensions) |
### Analysis
- Where Run A excelled: ...
- Where Run B excelled: ...
- Key improvement suggestion for Run A: ...
- Key improvement suggestion for Run B: ...
## Rules
- Score ONLY from evidence. If you can't determine a score from the artifacts, use N/A.
- Do NOT assume either side should win. Judge purely on what the diffs and logs show.
- Every score must cite specific evidence (file names, error counts, timing data).
- A higher agent count does NOT automatically mean better collaboration.
- A solo agent with effective sub-agent decomposition deserves full collaboration credit.
"""
)
Phase 4: Review Agent
Spawn a Review Agent to validate the evaluator's scoring. This catches bias,
scoring errors, and missed observations.
Agent(
subagent_type: "feature-dev:code-reviewer",
prompt: """
Review this development evaluation for accuracy and fairness.
## Original Task
[task-spec]
## Evaluator's Scorecard
[scorecard from Phase 3]
## Raw Evidence
- Run A diff: [summary]
- Run B diff: [summary]
## Check For
1. **Scoring accuracy** — Does each score match the justification?
2. **Evidence support** — Is every claim backed by the artifacts?
3. **Structural bias** — Does the rubric or framing unfairly advantage one side?
(e.g., scoring format advantages, asymmetric evidence availability)
4. **Missing observations** — Did the evaluator miss something important in the diffs?
5. **Actionable insights** — Are the improvement suggestions specific and implementable?
Output:
- APPROVED if the evaluation is fair and accurate
- REVISED SCORES if you disagree (with justification for each change)
- Key additional observations the evaluator missed
"""
)
Phase 5: Gap Report
After scoring, spawn a Gap Analysis Agent to identify systematic weaknesses
in the agent-worker team and propose concrete improvements.
Agent(
subagent_type: "general-purpose",
prompt: """
Analyze this development evaluation and produce a Gap Report for the
agent-worker team. Focus on root causes and actionable improvements.
## Scorecard
[scorecard from Phase 3/4]
## Event Log Highlights
[team-metrics.txt content]
## Report Structure
### Weakest Dimensions (ranked)
List dimensions where agent-worker scored lowest, with delta vs solo.
### Root Cause Analysis
For each weak dimension, identify WHY it underperformed:
- Prompt deficiency (agent didn't know what to do)
- Coordination overhead (multi-agent tax)
- Model capability gap (wrong model for the role)
- Infrastructure issue (tool failures, timeouts)
- Task mismatch (wrong tier for team approach)
### Improvement Recommendations
For each root cause, ONE specific change to the workspace YAML:
- Quote the exact line to change
- Show before/after
- Estimated impact (high/medium/low)
### Team Composition Assessment
Was this the right team for this task tier?
If not, suggest a leaner alternative configuration.
### Next Eval Priority
What single change would most improve the score if re-run?
"""
)
Save output to $EVAL_DIR/gap-report.md.
Phase 6: Final Report
Compile everything into the eval directory:
.agent-workspace/evals/YYYY-MM-DD-{slug}/
task-spec.md # The task definition
baseline-sha.txt # Git SHA before both runs
baseline-status.txt # Working tree status before runs
eval-workspace.yaml # Workspace config used
team-log.jsonl # Agent-worker event log (from parallel capture)
team-metrics.txt # eval-metrics.ts output
workspace-data/ # Persistent workspace data (from data_dir)
solo-report.md # Claude-code completion report
diff-team.patch # git diff from team worktree (clean)
diff-solo.patch # git diff from solo worktree (clean)
scorecard.md # Evaluator's scorecard (possibly revised)
review.md # Review agent's assessment
gap-report.md # Gap analysis and improvement recommendations
Update the running summary:
.agent-workspace/evals/SUMMARY.md
| Date | Task | Tier | Team | Solo | Delta | Config | Key Insight |
|------|------|------|------|------|-------|--------|-------------|
Notification Mechanism
How each side signals completion:
| Side | Mode | Signal | How to detect |
|---|
| Agent-Worker | aw run (task) | Process exits with status 0/1/2 | Bash run_in_background — notified on exit |
| Agent-Worker | service | EVAL_COMPLETE_TEAM in channel | aw read "@eval-team#general" 5 |
| Agent-Worker | service (backup) | All agents idle | agents MCP tool |
| Claude-Code | worktree | Agent tool returns | Natural — return value contains report |
| Evaluator | — | Agent tool returns | Parse scorecard from return |
| Reviewer | — | Agent tool returns | APPROVED or REVISED SCORES |
aw run is the preferred mode because:
- Exit code = completion status (0=done, 1=failed, 2=timeout)
--tag allows re-running the same config for iterations--wait 30m prevents indefinite hanging- No need for channel polling or explicit completion markers
Caveat: aw run auto-removes workspace on completion. Mitigated by:
data_dir in YAML config → persists workspace data to disk- Parallel
aw log @eval-team -f --json pipe → captures event stream before removal
- Git diff captured from team worktree → not affected by workspace removal
CLI Target Syntax Reference
The aw CLI uses a unified target syntax: [agent][@workspace[:tag]][#channel]
$AW_CMD send "claude-code@eval-team" "msg" # send to agent in workspace
$AW_CMD send "@eval-team" "msg" # send to workspace (kickoff channel)
$AW_CMD read "@eval-team#general" 10 # read workspace channel
$AW_CMD log @eval-team -f --json # stream workspace events (SSE)
$AW_CMD log @eval-team:iter-1 -f --json # stream tagged workspace events
See target.ts for full grammar.
Evaluation Dimensions
| Dimension | What to measure |
|---|
| Autonomy | Human interventions, stuck episodes, self-recovery |
| Quality | Correctness, test pass rate, review findings |
| Speed | Wall-clock time from start to completion |
| Maintainability | Code readability, structure, naming, test coverage for future changes |
| Completeness | Requirements coverage, edge cases handled |
| Cost | Total tokens/API calls consumed (N/A if unavailable) |
No pre-assumed advantages. The eval measures what actually happened, not what
the architecture theoretically enables. The whole point is to discover — with evidence
— whether and when the team approach outperforms solo.
Task Tiers
| Tier | Examples |
|---|
| T1 (simple) | Add CLI command, fix known bug, add config option |
| T2 (medium) | New feature 3-5 files, refactor with cross-cutting concerns |
| T3 (complex) | New subsystem, multi-package change, perf optimization |
Start with T2 — that's where the comparison is most interesting.
Dimension Weights by Tier
Different tiers weight dimensions differently. Apply multipliers when
computing weighted totals (optional — useful for trend analysis):
| Dimension | T1 weight | T2 weight | T3 weight |
|---|
| Autonomy | 1.0 | 1.5 | 2.0 |
| Quality | 2.0 | 2.0 | 2.0 |
| Speed | 1.5 | 1.0 | 0.5 |
| Maintainability | 0.5 | 1.5 | 2.0 |
| Completeness | 2.0 | 2.0 | 2.0 |
| Cost | 0.5 | 1.0 | 1.5 |
T1: Speed and quality matter most — overhead dominates.
T2: Balanced — autonomy and maintainability start to differentiate.
T3: Autonomy and maintainability dominate — system-level quality shows here.
Team Design Iteration
The eval isn't just about comparing scores — it's a feedback loop for finding the
ideal workspace team design: agent prompts, role division, team composition, and
coordination patterns.
What to iterate on
Each eval produces a scorecard + reviewer analysis. Extract specific improvement
actions for the workspace YAML:
| Signal in scorecard | What to change in YAML |
|---|
| Low Autonomy — agents got stuck, waited | Strengthen agent instructions: add self-unblocking rules, error recovery steps |
| Low Quality — bugs, missing tests | Add a dedicated reviewer agent, or strengthen review in orchestrator prompt |
| Low Speed — sequential when could be parallel | Restructure roles so more work can happen concurrently |
| Low Maintainability — messy code, missing tests | Add reviewer agent or strengthen review prompt; require test coverage in done criteria |
| Low Completeness — missed requirements | Add requirement-checking step in orchestrator's completion protocol |
| High Cost — too many tokens for the result | Reduce agent count, use cheaper models for simple roles, tighten prompts |
Team composition experiments
The workspace YAML is the experiment config. Try different team shapes:
agents:
claude-code: { runtime: claude-code, model: sonnet }
kimi-code: { runtime: ai-sdk, model: kimi-code:kimi-for-coding }
deepseek: { runtime: ai-sdk, model: deepseek:deepseek-chat }
agents:
claude-code: { runtime: claude-code, model: sonnet }
coder-1: { runtime: ai-sdk, model: deepseek:deepseek-chat }
coder-2: { runtime: ai-sdk, model: kimi-code:kimi-for-coding }
reviewer: { runtime: claude-code, model: haiku }
agents:
claude-code: { runtime: claude-code, model: sonnet }
coder: { runtime: ai-sdk, model: deepseek:deepseek-chat }
agents:
planner: { runtime: claude-code, model: opus }
coder-1: { runtime: ai-sdk, model: deepseek:deepseek-chat }
coder-2: { runtime: ai-sdk, model: deepseek:deepseek-chat }
reviewer: { runtime: claude-code, model: sonnet }
Agent prompt iteration
The instructions field in YAML is the most impactful lever. Key patterns to test:
Orchestrator prompt variables:
- Decomposition granularity: "break into 2-3 subtasks" vs "one subtask per agent"
- Review depth: "spot-check" vs "review every file changed"
- Completion protocol: strict (checklist) vs loose (judgment call)
- Communication style: verbose (explain reasoning) vs terse (action only)
Executor prompt variables:
- Autonomy level: "ask @orchestrator before making design decisions" vs "make best judgment, report after"
- Scope discipline: "ONLY do your assigned subtask" vs "fix related issues you notice"
- Reporting: "post progress updates" vs "only report when done"
A/B testing team designs
For rigorous comparison, run the same task with two different YAML configs:
$AW_CMD run eval-workspace-A.yaml --tag config-a --wait 30m
$AW_CMD run eval-workspace-B.yaml --tag config-b --wait 30m
Both run against the same task-spec.md. Compare the scorecards to see which team
design produces better results. Track in SUMMARY.md with a "Config" column.
Graduation criteria
A team design "graduates" when:
- It consistently outperforms claude-code solo on T2+ tasks (Delta > +3)
- It doesn't regress on T1 tasks (Delta >= -1, acceptable overhead)
- The cost ratio is reasonable (Team cost < 3x Solo cost for > +5 quality delta)
- Prompts are stable — minor rewording doesn't cause score swings > ±2
Iteration Loop
1. /dev-eval → run eval → get scorecard
2. Analyze: which dimension scored lowest? What caused it?
3. Iterate on ONE variable at a time:
- Agent prompts (instructions field)
- Team composition (add/remove/change agents)
- Role division (orchestrator vs executor split)
- Channel structure (general vs specialized channels)
- Model selection (which model for which role)
4. Re-run SAME task with new YAML config → measure delta
5. Track in SUMMARY.md with Config column → build trend data
6. When team design graduates → use it as the default workspace config
Each eval cycle should change exactly one variable for clean attribution.
Quick Start
1. Pick a T2 task
2. /dev-eval
3. I'll define the spec, create worktrees for both sides,
launch parallel runs, wait for completion,
run evaluator + reviewer, and save the report
4. Review scorecard → identify weakest dimension
5. Modify workspace YAML (one change) → re-run → compare
