challenge-run

name	challenge-run
description	Use after /challenge-gen when challenges are ready to execute. Dispatches the agent on each challenge and evaluates result, trap detection, and calibration. Keywords: challenge, run, evaluate, active-learning, training, agent, performance, calibration.
argument-hint	<agent-name>
disable-model-invocation	false
user-invocable	true
allowed-tools	Read, Write, Grep, Glob, Task, "Bash(git:*)", "Bash(mkdir:*)", "Bash(wc:*)"
context	inline

Challenge Run: Execute and Evaluate Agent Challenges

You are running challenge-run -- dispatching an agent on targeted challenges and evaluating performance. Target agent: $ARGUMENTS

When to Use

• After /challenge-gen produces a challenge set -- execute the challenges and measure agent performance
• When testing whether an agent has improved on previously identified weaknesses
• Before /active-learn to produce evaluation data for the learning loop
• When you want objective evidence of agent capability rather than speculation

Don't Use When

• No challenge files exist for the target agent — run /challenge-gen first to produce them
• Agent is not defined in .claude/team.yaml — the skill requires the agent's role, model, and ownership patterns from the team manifest
• You want to generate challenges rather than execute them — use /challenge-gen to create a challenge set first

Prerequisites

This skill requires the following infrastructure:

• challenge files — One or more challenge files in .claude/tackline/memory/agents/<agent>/challenges/*-challenges.md, generated by /challenge-gen (how to generate)
• team.yaml — Agent must be listed in .claude/team.yaml with role, model, and ownership patterns; these are injected into the agent's dispatch prompt (how to create)
• git repo with worktree support — Commit-replay challenges use isolation: "worktree" to check out the base_commit state without affecting the main working tree; the repo must support git worktree add (standard in any non-shallow clone)

If challenge files are missing, the skill stops immediately and directs the user to run /challenge-gen first. If team.yaml does not list the agent, the skill errors with the available agent names. If the repo does not support worktrees, commit-replay challenges will fail during dispatch — edge-case challenges will still run normally.

How It Works

Detect upstream /challenge-gen output or load challenge file
  -> Read agent learnings + team manifest
    -> For each challenge, dispatch agent with scenario + criteria (NOT trap/truth)
      -> Evaluate agent output against hidden trap + ground truth
        -> Score: result, trap detection, confidence calibration
          -> Write evaluation results to .claude/tackline/memory/agents/<name>/challenges/
            -> Emit pipe format evaluation report

---

Phase 0: Detect Input and Load Context

0a. Check for Upstream /challenge-gen Output

Search conversation context for the pipe-format pattern:

## Challenge Set: <agent-name>

**Source**: /challenge-gen

If found:

1. Read the upstream output. State: "Reading N items from /challenge-gen output above."
2. Extract: agent name, challenge items with scenarios, targeted weaknesses, difficulty levels
3. For each challenge, note its strategy (edge-case or commit-replay) and extract base_commit if present. Challenges with base_commit will use worktree isolation in Phase 1b.
4. Record the upstream **Pipeline** field for provenance

If not found:

1. If $ARGUMENTS provides an agent name, look for challenge files (Phase 0b)
2. If $ARGUMENTS is empty, list available agents from .claude/team.yaml and ask the user which to target. Stop and wait.

0b. Load Challenge File from Disk

If no upstream pipe-format output was detected, search for the most recent challenge file:

# Find challenge files for this agent
git ls-files -- ".claude/tackline/memory/agents/$ARGUMENTS/challenges/*-challenges.md"

Also check untracked files with Glob: .claude/tackline/memory/agents/<name>/challenges/*-challenges.md

Select the most recent file (by timestamp in filename). Read it and extract all challenges with their full details: Scenario, Acceptance Criteria, Hidden Trap, Ground Truth. For commit-replay challenges, also extract the Base commit field. Note which challenges have a base_commit and which don't — this determines isolation mode in Phase 1b.

If no challenge file exists, tell the user to run /challenge-gen <agent-name> first and stop.

0c. Read Agent Context

1. Read .claude/team.yaml and confirm the agent name exists in the members list
2. Extract the agent's role, model, and owns patterns
3. Read .claude/tackline/memory/agents/<name>/learnings.md if it exists

0d. Prerequisite Gate

Do not proceed until:

• Challenge set is loaded (from context or file) with at least 1 challenge
• Agent exists in team.yaml with role and model known
• Agent learnings read (or confirmed absent for new agents)

---

Phase 1: Dispatch Challenges

Process challenges serially -- each dispatch is independent but serial execution avoids API throttling and lets you build a complete evaluation picture.

1a. Compose the Challenge Prompt

For each challenge, build a self-contained prompt for the target agent. The prompt MUST include:

1. Agent identity: Name, role, ownership areas (from team.yaml)
2. Agent learnings: Full contents of their learnings.md
3. Challenge scenario: The scenario section from the challenge definition
4. Acceptance criteria: The criteria section from the challenge definition
5. Reflection protocol: Ask the agent to self-assess at the end

CRITICAL: Never include the Hidden Trap or Ground Truth in the agent prompt. These are for evaluation only. Including them defeats the purpose of the challenge.

For commit-replay challenges with base_commit: Prepend a setup step to the prompt instructing the agent to run git checkout <base_commit> as their first action. This resets the worktree to the pre-fix state so the agent works against the actual codebase at the time of the bug.

The prompt structure:

You are , a on the team.

Setup (commit-replay only)

Run this command before anything else to reset to the pre-fix codebase state:

git checkout <base_commit>

Your Learnings

Task

Acceptance Criteria

Reflection

After completing the task, provide:

1. Approach: What approach did you take and why?
2. Confidence: How confident are you in your solution? (LOW / MEDIUM / HIGH)
3. Uncertainty: What aspects are you least sure about?
4. Alternative approaches: What other approaches did you consider and why did you reject them?
5. Risk areas: Where might this solution break or be wrong?

1b. Dispatch

For commit-replay challenges with base_commit: Use worktree isolation so the agent gets its own repo copy and can check out the pre-fix state without affecting the main working tree.

Task({
  subagent_type: "general-purpose",
  isolation: "worktree",
  model: "<agent's model from team.yaml>",
  prompt: "<composed prompt from 1a>"
})

For edge-case challenges (no base_commit): Dispatch without worktree isolation (existing behavior).

Task({
  subagent_type: "general-purpose",
  model: "<agent's model from team.yaml>",
  prompt: "<composed prompt from 1a>"
})

Use serial dispatch (no run_in_background). Each challenge is independent but evaluation benefits from sequential processing.

1c. Capture Output

Store the agent's full response for evaluation in Phase 2. Extract:

• The agent's solution/implementation
• Their self-reported confidence level
• Their identified risk areas and uncertainties
• Their reasoning about approach selection

For commit-replay challenges with worktree isolation: Also capture the agent's git diff from the worktree. The Task tool output from a worktree-isolated agent includes the diff of changes made in the worktree. Record this diff as the Agent Diff — it shows exactly what the agent changed and becomes primary evaluation material in Phase 2a for precise comparison against the ground truth.

---

Phase 2: Evaluate Each Challenge

After each dispatch returns, evaluate the agent's output against the hidden evaluation criteria.

2a. Result Assessment (PASS / PARTIAL / FAIL)

Check the agent's output against the acceptance criteria:

PASS: All acceptance criteria are met. The solution is correct and complete.

PARTIAL: Some acceptance criteria are met but not all. The solution works for the main case but misses edge cases or has minor issues.

FAIL: The solution does not meet the primary acceptance criteria. Fundamental misunderstanding or incorrect approach.

Objective checks (prefer these over subjective assessment):

• For commit-replay challenges with an Agent Diff (captured in Phase 1c): Compare the agent's actual diff against the ground truth diff. Do they touch the same files? Do they address the same root cause? Is the fix semantically equivalent even if not line-identical? The Agent Diff provides precise, objective evidence — prefer it over assessing the agent's prose description of their changes.
• For commit-replay challenges without an Agent Diff: Fall back to comparing the agent's described approach against the ground truth diff direction.
• For edge-case challenges: Does the agent's solution handle the specific edge case described in the hidden trap?
• If the challenge involves code: Does the approach match the ground truth direction?

2b. Trap Detection (CAUGHT / PARTIAL / MISSED)

Evaluate whether the agent recognized the non-obvious difficulty:

CAUGHT: The agent explicitly identified the trap or the core difficulty. Their solution handles it correctly. Evidence: they mentioned it in their reasoning, risk areas, or uncertainties.

PARTIAL: The agent showed awareness of something being tricky in the right area but did not fully articulate the trap or only partially handled it. Evidence: their uncertainty or risk areas point in the right direction.

MISSED: The agent showed no awareness of the trap. Their solution fails on the trap scenario. Evidence: no mention in reasoning, high confidence despite the trap, solution breaks on the trap case.

2c. Confidence Calibration (OVER / UNDER / WELL_CALIBRATED)

Compare the agent's self-reported confidence against their actual performance:

Self-reported	Actual result	Calibration
HIGH	PASS	WELL_CALIBRATED
HIGH	PARTIAL	OVER
HIGH	FAIL	OVER
MEDIUM	PASS	UNDER (slightly)
MEDIUM	PARTIAL	WELL_CALIBRATED
MEDIUM	FAIL	OVER
LOW	PASS	UNDER
LOW	PARTIAL	WELL_CALIBRATED
LOW	FAIL	WELL_CALIBRATED

Nuance: If the agent reported HIGH confidence but also correctly identified the trap in their risk areas, lean toward WELL_CALIBRATED -- they knew the risk even if their headline confidence was high.

2d. Reasoning Quality Assessment

Briefly assess the quality of the agent's reasoning process:

• Did they explore multiple approaches before committing?
• Did they identify relevant constraints and trade-offs?
• Was their rejection of alternative approaches well-reasoned?
• Did they demonstrate domain knowledge appropriate to their role?

2e. Growth Signal

Synthesize what this result tells us about the agent's capability relative to the targeted weakness:

• Strength emerging: The agent is developing competence in this area (PASS + CAUGHT)
• Awareness growing: The agent recognizes the difficulty but cannot yet solve it (PARTIAL + PARTIAL/CAUGHT)
• Blind spot persists: The agent does not recognize the difficulty (any + MISSED)
• Overcorrection risk: The agent is cautious in this area but their caution is misplaced (PASS + CAUGHT + UNDER)

---

Phase 3: Persist Results

3a. Write Evaluation File

mkdir -p .claude/tackline/memory/agents/<name>/challenges

Write evaluation results to .claude/tackline/memory/agents/<name>/challenges/<timestamp>-evaluation.md:

# Challenge Evaluation: <agent-name>

Evaluated: <date>
Challenge source: <challenge file path or "pipe-format context">
Pipeline: <upstream pipeline> -> /challenge-run (N evaluated)

## Challenge 1: <title>

**Result**: PASS | PARTIAL | FAIL
**Trap Detection**: CAUGHT | PARTIAL | MISSED
**Confidence Calibration**: OVER | UNDER | WELL_CALIBRATED
**Targeted Weakness**: <weakness name>
**Difficulty**: <level>

### Agent's Approach

<Brief summary of what the agent did>

### Agent Diff *(commit-replay only)*

```diff
<git diff captured from the agent's worktree — omit this section for edge-case challenges>

Evaluation Notes

<Why this result/trap/calibration rating was assigned. Cite specific evidence from the agent's output. For commit-replay challenges, reference specific lines from the Agent Diff compared against the ground truth.>

Growth Signal

<What this tells us about the agent's trajectory on this weakness>

---

Challenge 2: ...

---

Overall Assessment

• Results: X/N passed, Y partial, Z failed
• Trap detection: X caught, Y partial, Z missed
• Calibration: X well-calibrated, Y over-confident, Z under-confident
• Key insight:

---

## Phase 4: Emit Pipe Format Output

Output the evaluation report in pipe format for downstream consumption by /active-learn:

```markdown
## Challenge Results: <agent-name>

**Source**: /challenge-run
**Input**: <agent-name>
**Pipeline**: <upstream pipeline> -> /challenge-run (N evaluated)

### Items (N)

1. **<challenge-title>: <PASS|PARTIAL|FAIL>** — <one-line summary of agent performance>
   - targeted_weakness: <from challenge>
   - result: PASS | PARTIAL | FAIL
   - trap_detection: CAUGHT | PARTIAL | MISSED
   - confidence_calibration: OVER | UNDER | WELL_CALIBRATED
   - reasoning_quality: <brief assessment>
   - growth_signal: <what this result reveals>

2. ...

### Evaluation Summary

| Challenge | Result | Trap | Confidence | Growth Signal |
|-----------|--------|------|------------|---------------|
| [title]   | [P/F]  | [C/P/M] | [O/U/W] | [signal] |

Overall: [X/N passed], [trap detection rate], [calibration assessment]

### Summary

<One paragraph synthesizing what these results reveal about the agent's growth trajectory, which weaknesses are improving, which persist, and what the next training cycle should focus on.>

---

Guidelines

1. Never leak evaluation criteria to the agent. The Hidden Trap and Ground Truth sections exist solely for post-hoc evaluation. Including them in the dispatch prompt invalidates the challenge. This is the single most important constraint.
2. Objective over subjective. Prefer concrete checks (diff comparison, edge case handling, code correctness) over subjective quality assessments. When objective checks are available, use them first and let subjective assessment fill gaps.
3. Serial dispatch. Process one challenge at a time. This avoids API throttling and produces cleaner evaluation flow. Unlike blossom spikes, challenges are not independent explorations -- they are controlled experiments.
4. Calibration matters as much as correctness. An agent that fails but knows it will fail (LOW confidence + FAIL) is better calibrated than one that fails confidently (HIGH confidence + FAIL). Calibration predicts future reliability.
5. Growth signals over grades. The goal is not to score the agent but to understand their trajectory. "Blind spot persists" is more actionable than "FAIL." The evaluation feeds /active-learn, not a report card.
6. Graceful degradation. No upstream pipe format? Read from challenge files. No ground truth diff? Use acceptance criteria only. No learnings file? Dispatch without learnings. Always produce something useful.
7. Evidence in evaluation notes. Every rating must cite specific evidence from the agent's output. "MISSED" is not useful without "the agent did not mention X despite it being the core difficulty."
8. Commit-replay fidelity. For commit-replay challenges, compare the agent's approach direction against the ground truth diff. Semantic equivalence counts -- the agent does not need to produce an identical diff, just address the same root cause with a sound approach.
9. Self-contained agent prompts. The dispatched agent cannot read the skill file or access parent context. Everything the agent needs (identity, learnings, scenario, criteria, reflection protocol) must be in the prompt.
10. Results persist for longitudinal tracking. Evaluation files accumulate in the challenges directory so /active-learn can track improvement over multiple cycles.
11. Worktree isolation for commit-replay fidelity. Worktrees give each agent an isolated copy of the repository, so commit-replay challenges get authentic pre-fix state without affecting the main working tree. The agent checks out base_commit inside the worktree and works against the actual codebase at the time of the bug. Edge-case challenges do not need worktrees since they operate on the current codebase state.

See also: /challenge-gen (upstream producer), /diagnose-agent (struggle profiling), /active-learn (downstream consumer).