agent-eval-loop

name	agent-eval-loop
description	Run and improve the c0 agent evaluation loop using the local and remote Cloudflare MCP harness, versioned history notes, and current Cloudflare product updates. Use when evaluating agent quality, harness behavior, retrieval quality, model/tool performance, or when the user asks to benchmark, score, improve, or retest the agent workflow.

Agent Eval Loop

Purpose

Use this skill to run repeatable agent evaluations, compare outcomes over time, apply targeted improvements, retest, and record versioned results in notes/agent-eval-history/.

Start Here

1. Read every versioned note in notes/agent-eval-history/ before proposing a new improvement.
2. Extract:
   • what worked
   • what regressed behavior
   • what has already been tried
   • what still looks promising
3. Do not repeat failed ideas unless you have new evidence that changes the hypothesis.

Pick An Eval Target

List available eval targets first:

nub run agent:eval:targets

Use --target <id> to inspect one target:

nub run agent:eval:targets --target linea-coordinator-runbooks

Prefer:

• configured-ai-search for narrow retrieval quality and /mcp behavior
• ai-search-mcp-regression for server, config, and route regressions
• agent-investigation-loop for broader multi-step workflows

Local Versus Remote

• nub run agent:eval:local runs the real Worker and /mcp route with a deterministic AI mock.
• nub run agent:eval:remote runs the real Worker and live Cloudflare retrieval path.
• Treat local runs as harness correctness signals.
• Treat remote runs as live quality signals.
• Never treat a passing local run as proof that live retrieval or live agent quality is correct.

Required Metrics

Score every eval on a 1-5 scale:

• relevance
• helpfulness
• correctness

Also record:

• taskDurationMs
• local run duration when available
• remote run duration when available

Rubric:

• 1: unusable or misleading
• 2: weak, incomplete, or mostly off-target
• 3: acceptable but missing important value
• 4: strong and useful with minor issues
• 5: highly effective, correct, and ready to trust

Web Research

Each eval loop should search the web for recent Cloudflare changes that may improve the harness or implementation:

• Agents SDK
• AI Search
• Vectorize
• Workers AI
• MCP- and Workers-related testing improvements

Capture:

• title
• URL
• why it matters for the current eval target

Core Loop

1. Read prior notes and summarize the important lessons.
2. Choose a target and define success criteria.
3. Run the relevant local and/or remote harness commands.
4. Review outputs and score the metrics.
5. If the user wants improvement work, make the smallest high-signal change that addresses the current hypothesis.
6. Retest using the same target.
7. Compare metrics and raw evidence.
8. Record a versioned history note.

Use The Helper Scripts

Create a machine-readable summary first:

nub run agent:eval:metrics --input /tmp/agent-eval-summary.json --output /tmp/agent-eval-summary.json

Preview the next note path:

nub run agent:eval:note:next --recap "bootstrap-agent-eval-workflow"

Write the versioned note and artifact:

nub run agent:eval:note:write --input /tmp/agent-eval-summary.json --recap "bootstrap-agent-eval-workflow"

Summary Payload Expectations

The input JSON for agent:eval:metrics should include:

• targetId
• taskSummary
• workflowType
• recap
• local
• remote
• metrics
• baselineObservations
• changesAttempted
• worked
• didntWork
• nextHypotheses
• webUpdates
• rawEvidence
• existingHistoryReviewed
• sourceCommands

See reference.md for a complete example payload and note shape.

Output Requirements

For each eval run, produce:

• a concise summary of findings
• exact commands run
• the metric scores
• whether the result came from local or remote harness coverage
• the note path written under notes/agent-eval-history/
• the next best hypothesis

Improvement Guidance

• Prefer one focused change per loop when validating a hypothesis.
• Keep the before/after comparison tight.
• If a change helps locally but not remotely, say so explicitly.
• If a Cloudflare product update suggests a new approach, reference it in both the write-up and the next hypothesis.

Stop Conditions

Stop the loop when:

• the user only asked for evaluation, not changes
• the remote run is blocked by missing credentials or infrastructure access
• the next change would be speculative without stronger evidence
• you have already reached a clear conclusion for the current target

Manual Invocation

Use .cursor/commands/run-agent-eval.md for a repeatable chat entrypoint.