agent-qa

name	agent-qa
description	End-to-end QA testing for Agent2 agents. Starts the agent service, sends real API requests from eval datasets, validates responses against the output schema, and generates a health report with confidence stats. Use when asked to "test this agent", "QA the agent", "does this agent work", or "run agent health check".

Agent QA

End-to-end testing for a deployed or locally running Agent2 agent.

When to Use

• After generating a new agent with /brain-clone or /creating-agents
• After changing an agent's prompt, tools, or schema
• Before shipping an agent to production
• When debugging unexpected agent behavior

Workflow

Step 1: Identify the agent

Ask which agent to test if not obvious from context:

ls agents/*/config.yaml 2>/dev/null | sed 's|agents/||;s|/config.yaml||'

Step 2: Load test cases

Check for existing eval datasets:

AGENT_NAME="<agent-name>"
EVAL_FILE="tests/promptfoo/$AGENT_NAME/dataset.json"
if [ -f "$EVAL_FILE" ]; then
  echo "EVAL_FOUND: $EVAL_FILE"
  cat "$EVAL_FILE"
else
  echo "NO_EVAL_DATASET"
fi

If NO_EVAL_DATASET: generate test cases from the agent's example cases in config.yaml and the prompt in agent.py. Create at least 3 cases:

1. A case that should return the primary complete outcome
2. An empty/incomplete case that should trigger needs_clarification
3. A case with a defect that should trigger rejected (if applicable)

Step 3: Start the agent

# Check if already running
AGENT_PORT=$(grep "port:" "agents/$AGENT_NAME/config.yaml" | awk '{print $2}')
curl -s "http://localhost:$AGENT_PORT/health" 2>/dev/null && echo "ALREADY_RUNNING" || echo "NEEDS_START"

If NEEDS_START:

uv run agent2 serve "$AGENT_NAME" &
AGENT_PID=$!
sleep 3
curl -s "http://localhost:$AGENT_PORT/health"

Step 4: Run test cases

For each test case, send a real POST /tasks request:

curl -s -X POST "http://localhost:$AGENT_PORT/tasks?mode=sync" \
  -H "Authorization: Bearer dev-token-change-me" \
  -H "Content-Type: application/json" \
  -d '{"input": <test-case-input>}'

For each response, validate:

1. HTTP status is 200
2. Response contains "status": "completed"
3. result is present and contains the expected schema fields
4. result.status matches expected outcome (if specified in test case)
5. result.confidence is a number between 0 and 1
6. result.reasoning is non-empty
7. result.review_steps is a non-empty list
8. Schema consistency: no contradictory fields (e.g., complete + rejection_reason)

Step 5: Generate health report

AGENT QA REPORT: {agent-name}
═══════════════════════════════════════

Tests run:     {total}
Passed:        {passed}
Failed:        {failed}

| # | Case | Expected | Got | Confidence | Pass |
|---|------|----------|-----|------------|------|
| 1 | ... | complete | complete | 0.87 | ✓ |
| 2 | ... | clarification | clarification | 0.65 | ✓ |
| 3 | ... | rejected | complete | 0.45 | ✗ |

Confidence stats:
  Mean:   {mean}
  Min:    {min}
  Max:    {max}
  <0.85:  {count} ({pct}%)

Issues found:
  - {issue description}

Recommendation: {SHIP / FIX_BEFORE_SHIP / NEEDS_WORK}

Step 6: Check learnings

If the agent has operational learnings, show them:

LEARN_FILE="$HOME/.agent2/learnings/$AGENT_NAME.jsonl"
if [ -f "$LEARN_FILE" ]; then
  echo "LEARNINGS:"
  tail -10 "$LEARN_FILE"
else
  echo "No learnings yet (agent hasn't processed real cases)"
fi

Step 7: Cleanup

If we started the agent in Step 3:

kill $AGENT_PID 2>/dev/null

Completion Status

Report status as:

• PASS — all tests passed, confidence healthy
• PASS_WITH_CONCERNS — tests passed but confidence is low or edge cases untested
• FAIL — test failures found, list what's broken
• BLOCKED — agent couldn't start or API unreachable