agent-evaluation-framework

# Agent Evaluation Framework Workflow Use this skill to orchestrate evaluation sessions for subagents, identify procedural bottlenecks, and iteratively refine system prompts and capabilities utilizing Swarm intelligence principles. ## 0. Preparation - **Subagent Isolation**: Ensure that subagents spawned for evaluation do NOT utilize existing session brains or previous task knowledge. This is critical to maintain the integrity of meta-testing. - **Worktree Pre-creation**: Create isolated git worktrees using `agents/scripts/create_worktree.sh <task_id>` for each test case beforehand. Report where the worktrees were created to the user. Inside worktrees, builds MUST use the `tools/dev/gm.py` tool INSIDE the worktree. `gm.py` will automatically run `setup_worktree_build.py` to prepare the symlinks; manual execution of `setup_worktree_build.py` is not required. - **Test Injection**: Copy the target test case into the worktree (e.g., `test/mjsunit/repro.js`). - **Remote Compilation**: Ensure worktrees are set up to compile remotely (`use_remoteexec = true` in `args.gn`) before proceeding. ## 1. Core Directives - **Zero Hallucination**: Do not assume a test passes or fails without executing it. - **Worktree Enforcement**: Agents MUST operate strictly within their assigned worktree. They should NOT know the main V8 root exists. - **Test Scope**: Meta-refinement ALWAYS uses the tests in `agent-meta-tests` only. - **Test Immutability**: The `agent-meta-tests` directory cannot be changed. - **Crash Verification**: Only work on test-cases that still crash. - **Auto-Run Enforcement**: ALWAYS use `SafeToAutoRun: true` for ALL commands executed during meta-refinement. Approval must NEVER be asked of the user. - **Immediate Termination**: Terminate any agent immediately if it modifies the main V8 repository. ## 2. Agent Orchestration & Lifecycle Management - **Workspace Isolation**: Ensure agents are initialized in dedicated worktrees. - **Communication Routing**: Facilitate communication between sibling agents. Since evaluated agents operate independently, the Orchestrator/Main Agent must act as a message broker to share relevant findings and prevent duplicate work. - **User Reporting**: Synthesize high-level progress from all evaluated agents and keep the user informed without exposing raw logs or requiring manual approvals. ## 3. Evaluation & Divergence Analysis - **Entry Point**: A list of historical V8 fixes and their associated reproducing scripts (e.g., from `test/mjsunit/` or Buganizer). - **Execution**: Initialize the agent in an isolated worktree checked out to the **parent commit** of the target fix. Copy the repro script and command the agent to resolve the bug. - **Comparison**: Upon completion, compare the agent's proposed fix with the actual historical fix. - **Analysis**: If the solutions diverge: - Identify where the agent's reasoning deviated from the required fix. - Scan for "hallucinated complexity"—parts of the fix that were not logically required by the root cause but were added by the agent. - Evaluate if the agent overlooked critical architectural invariants or spec requirements. - **Hasty Fix Detection**: Specifically check if the agent's solution simply disabled an optimization or feature mistakenly instead of addressing the logic error. - **Root Cause Tracing**: Manually trace the logical steps required to reach the the *correct* historical fix. Identify the exact moment/decision where the agent chose a shallow path over a deep one. ## 4. Iterative Process Refinement & Skepticism The ultimate goal of evaluation is to harden the agent's skepticism and reasoning depth: - **Architectural Skepticism**: Require subagents to explicitly argue *against* a proposed fix before accepting it. Look at the problem from multiple orthogonal angles. - **Mandatory Deep Reasoning**: If a fix feels "guessed" or lacks direct evidence from GDB/Spec logs, spawn a subagent to reason deeper about the specific invariant being violated. - **Skill Updates**: Every evaluation session MUST conclude with a diff for relevant subsystem skills to bake in the lessons learned and prevent future failures. - **analyze_brain.py**: Scans agent logs for markers of shortcutting, logic failures, or divergence in reasoning.