agentifind-benchmark

name	agentifind-benchmark
description	Create a benchmark to measure CODEBASE.md effectiveness. Sets up hooks to run two parallel agents (one with guide, one without) and compare their efficiency. Requires /agentifind to be run first.

Agentifind Benchmark Setup

This skill creates a benchmark infrastructure to measure how effectively CODEBASE.md helps AI agents navigate your codebase.

Prerequisites

• .claude/CODEBASE.md must exist (run /agentifind first)
• .claude/codebase.json must exist

Procedure

Step 1: Verify Prerequisites

Check that required files exist:

test -f .claude/CODEBASE.md && test -f .claude/codebase.json && echo "Ready" || echo "Missing"

If files don't exist:

CODEBASE.md not found. Please run /agentifind first to generate the codebase guide, then run /agentifind-benchmark again.

Exit the skill if prerequisites are not met.

Step 2: Detect Repo Type and Tech Stack

Read .claude/codebase.json and check the repo_type field:

If repo_type is "terraform" (or other IaC types):

• This is an infrastructure repository
• Use the Infrastructure Benchmark Template (Step 6B)
• Focus on resource navigation, blast radius, dependency tracing

If repo_type is missing or not an IaC type:

• This is an application code repository
• Use the Application Benchmark Template (Step 6A)
• Identify:
  • Primary language (Python, TypeScript, JavaScript, Go)
  • Framework (FastAPI, Django, Flask, React, Next.js, Express, etc.)
  • Key patterns (async jobs, API routes, components, services)

This information will be used to create relevant benchmark tasks.

Step 3: Create Hooks Directory

mkdir -p .claude/hooks

Step 4: Create Hook Scripts

Create these 5 executable scripts in .claude/hooks/:

4.1: enforce-guide-restriction.sh

This hook blocks AGENT_WITHOUT_GUIDE from reading CODEBASE.md and logs violations:

#!/bin/bash
#
# Enforce Guide Restriction Hook
# Blocks AGENT_WITHOUT_GUIDE from reading CODEBASE.md or .claude/ files
# Logs all violation attempts
#

INPUT=$(cat)

TOOL_NAME=$(echo "$INPUT" | jq -r '.tool_name // empty')
TOOL_INPUT=$(echo "$INPUT" | jq -c '.tool_input // {}')
AGENT_TRANSCRIPT=$(echo "$INPUT" | jq -r '.agent_transcript_path // empty')

LOG_DIR="$(pwd)"
VIOLATIONS_FILE="$LOG_DIR/benchmark_violations.jsonl"

# Only check Read tool
if [[ "$TOOL_NAME" != "Read" ]]; then
    echo '{"decision":"approve"}'
    exit 0
fi

# Only check subagents with transcript
if [[ -z "$AGENT_TRANSCRIPT" || ! -f "$AGENT_TRANSCRIPT" ]]; then
    echo '{"decision":"approve"}'
    exit 0
fi

# Extract agent identifier from transcript (try multiple patterns)
TRANSCRIPT_CONTENT=$(cat "$AGENT_TRANSCRIPT" 2>/dev/null)

# Try to find AGENT_WITHOUT_GUIDE in various message structures
IS_WITHOUT_GUIDE=
   | grep -q ; 
    IS_WITHOUT_GUIDE=


 [[  !=  ]]; 
     
     0



FILE_PATH=$(  | jq -r )

   | grep -qiE ; 
    
    TIMESTAMP=$( )
    jq -n \
        --arg ts  \
        --arg agent  \
        --arg file  \
        --arg action  \
         \
        >> 

     
     0


 
 0

4.2: log-benchmark-tools.sh

This hook logs all tool calls for analysis with robust agent detection:

#!/bin/bash
#
# Benchmark Tool Call Logger
# Logs every tool call with agent identification
#

INPUT=$(cat)

TOOL_NAME=$(echo "$INPUT" | jq -r '.tool_name // empty')
TOOL_INPUT=$(echo "$INPUT" | jq -c '.tool_input // {}')
TOOL_USE_ID=$(echo "$INPUT" | jq -r '.tool_use_id // empty')
AGENT_TRANSCRIPT=$(echo "$INPUT" | jq -r '.agent_transcript_path // empty')
SESSION_ID=$(echo "$INPUT" | jq -r '.session_id // empty')

AGENT_TYPE="main"
AGENT_NAME="main"
TRANSCRIPT_FILE=""

if [[ -n "$AGENT_TRANSCRIPT" && -f "$AGENT_TRANSCRIPT" ]]; then
    AGENT_TYPE="subagent"
    TRANSCRIPT_FILE="$AGENT_TRANSCRIPT"

    # Read full transcript content for pattern matching
    TRANSCRIPT_CONTENT=$(cat "$AGENT_TRANSCRIPT" 2>/dev/null)

    # Robust detection: search entire transcript for agent markers
    if echo "$TRANSCRIPT_CONTENT" | grep -q "AGENT_WITH_GUIDE"; then
        AGENT_NAME=
       | grep -q ; 
        AGENT_NAME=
    
        
        AGENT_NAME=$(  .jsonl 2>/dev/null ||  )
    


TIMESTAMP=$( )
LOG_DIR=
JSONL_FILE=


FILE_PATH=
 [[  ==  ||  ==  ||  ==  ]]; 
    FILE_PATH=$(  | jq -r )


jq -n \
    --arg ts  \
    --arg   \
    --arg name  \
    --arg tool  \
    --argjson input  \
    --arg   \
    --arg session  \
    --arg transcript  \
    --arg file  \
     \
    >> 

 0

4.3: analyze-benchmark.sh

#!/bin/bash
#
# Analyze Benchmark Results
# Comprehensive analysis of tool calls, violations, and efficiency
#

LOG_DIR="$(pwd)"
LOG_FILE="$LOG_DIR/benchmark_log.jsonl"
VIOLATIONS_FILE="$LOG_DIR/benchmark_violations.jsonl"
TURNS_FILE="$LOG_DIR/benchmark_turns.jsonl"

if [[ ! -f "$LOG_FILE" ]]; then
    echo "No benchmark log found at: $LOG_FILE"
    echo "Run benchmark tasks first."
    exit 1
fi

echo "# Benchmark Analysis"
echo "Generated: $(date)"
echo ""

# Total entries
TOTAL=$(wc -l < "$LOG_FILE" | tr -d ' ')
echo "**Total logged tool calls:** $TOTAL"
echo ""

echo "## Tool Calls by Agent"
echo "| Agent | Count |"
echo "|-------|-------|"

for AGENT in "main" "WITH_GUIDE" ; 
    COUNT=$(grep   |  -l |  -d )
    [[  -gt 0 ]] &&  


 


 
 

 AGENT   ; 
    COUNT=$(grep   |  -l |  -d )
     [[  -gt 0 ]]; 
         
         
         
        grep   | \
            jq -r  |  |  -c |  -rn | \
              count tool; 
                 
            
         
    



 
 [[ -f  ]]; 
    VIOLATION_COUNT=$( -l <  |  -d )
     
     
     [[  -gt 0 ]]; 
         
         
        jq -r   | \
             IFS=$  ts agent file; 
                 
            
    

     

 


 [[ -f  ]]; 
     
     
     
     AGENT   ; 
        TURNS=$(grep   |  -l |  -d )
        [[  -gt 0 ]] &&  
    
     



 
 

WITH=$(grep   |  -l |  -d )
WITHOUT=$(grep   |  -l |  -d )

 [[  -gt 0 &&  -gt 0 ]]; 
    RATIO=$(awk )
    REDUCTION=$(awk )

     
     
     
     
     
     

     
     
     


 


 
 

 AGENT   ; 
    COUNT=$(grep   |  -l |  -d )
     [[  -gt 0 ]]; 
         
         
        grep   | \
            jq -r  | \
             -u |  -20
         
         
    

4.4: track-turns.sh

PostToolUse hook to track agent turns (API round-trips):

#!/bin/bash
#
# Track Agent Turns (PostToolUse)
# Logs completed tool calls to measure turns/round-trips
#

INPUT=$(cat)

TOOL_NAME=$(echo "$INPUT" | jq -r '.tool_name // empty')
TOOL_USE_ID=$(echo "$INPUT" | jq -r '.tool_use_id // empty')
AGENT_TRANSCRIPT=$(echo "$INPUT" | jq -r '.agent_transcript_path // empty')

# Only track subagent activity
if [[ -z "$AGENT_TRANSCRIPT" || ! -f "$AGENT_TRANSCRIPT" ]]; then
    exit 0
fi

# Detect agent type
TRANSCRIPT_CONTENT=$(cat "$AGENT_TRANSCRIPT" 2>/dev/null)
AGENT_NAME="unknown"

if echo "$TRANSCRIPT_CONTENT" | grep -q "AGENT_WITH_GUIDE"; then
    AGENT_NAME="WITH_GUIDE"
elif echo "$TRANSCRIPT_CONTENT" | grep -q "AGENT_WITHOUT_GUIDE"; then
    AGENT_NAME="WITHOUT_GUIDE"
fi

# Skip if not a benchmark agent
if [[ "$AGENT_NAME" ==  ]]; 
     0


TIMESTAMP=$( )
LOG_DIR=
TURNS_FILE=

jq -n \
    --arg ts  \
    --arg name  \
    --arg tool  \
    --arg   \
     \
    >> 

 0

4.5: reset-benchmark.sh

#!/bin/bash
#
# Reset all benchmark logs
#

LOG_DIR="$(pwd)"

rm -f "$LOG_DIR/benchmark_log.jsonl"
rm -f "$LOG_DIR/benchmark_log.md"
rm -f "$LOG_DIR/benchmark_violations.jsonl"
rm -f "$LOG_DIR/benchmark_turns.jsonl"
rm -f "$LOG_DIR/benchmark_results.md"

echo "Benchmark logs cleared:"
echo "  - benchmark_log.jsonl"
echo "  - benchmark_violations.jsonl"
echo "  - benchmark_turns.jsonl"
echo ""
echo "Ready for new benchmark run."

Make all scripts executable:

chmod +x .claude/hooks/*.sh

Step 5: Create settings.json

Create .claude/settings.json (or merge with existing):

{
  "hooks": {
    "PreToolUse": [
      {
        "hooks": [
          {"type": "command", "command": ".claude/hooks/log-benchmark-tools.sh"}
        ]
      },
      {
        "matcher": "Read",
        "hooks": [
          {"type": "command", "command": ".claude/hooks/enforce-guide-restriction.sh"}
        ]
      }
    ],
    "PostToolUse": [
      {
        "hooks": [
          {"type": "command",  
        
      
    
  

Step 6: Create Benchmark Tasks

Based on the repo type detected in Step 2, use the appropriate template.

---

Step 6A: Application Benchmark Template

Use this template when repo_type is NOT "terraform" or similar IaC types.

Create .claude/BENCHMARK_ENGINEERING.md with 7 tasks tailored to the detected tech stack.

Task Categories:

1. TASK-1 (⭐ Fundamental): Find a configuration value
2. TASK-2 (⭐ Fundamental): Locate a specific function/class
3. TASK-3 (⭐⭐ Intermediate): Trace a data flow
4. TASK-4 (⭐⭐⭐ Advanced): Add a new feature following existing patterns
5. TASK-5 (⭐⭐⭐ Advanced): Fix a bug with provided symptoms
6. TASK-6 (⭐⭐⭐⭐ Expert): Debug a race condition or performance issue
7. TASK-7 (⭐⭐⭐⭐⭐ Expert): Design a cross-cutting feature

Template:

# {Project Name} Engineering Benchmark

**Purpose:** Measure CODEBASE.md effectiveness on real engineering tasks.

---

## Orchestrator Instructions

You are running a benchmark to compare agent efficiency WITH vs WITHOUT the CODEBASE.md guide.

**IMPORTANT RULES:**
1. Run agents in FOREGROUND (do NOT use run_in_background)
2. Run both agents IN PARALLEL for the same task (single message, two Task calls)
3. Wait for both to complete before reporting results
4. Report token usage from the Task tool response if available
5. Track metrics from agent self-reports

---

## How to Run

### 1. Reset Logs
\`\`\`bash
.claude/hooks/reset-benchmark.sh
\`\`\`

### 2. Run Both Agents in Parallel (Foreground)

For each task, spawn BOTH subagents in a SINGLE message using two Task tool calls.
Do NOT use run_in_background - we need the full response including token usage.

\`\`\`
Task 1 (WITH_GUIDE):
- subagent_type: "general-purpose"
- run_in_background: false  # IMPORTANT: foreground for token tracking
- prompt: |
    AGENT_WITH_GUIDE: BENCHMARK

    You have access to .claude/CODEBASE.md - a navigation guide for this codebase.

    WORKFLOW:
    1. Read .claude/CODEBASE.md FIRST
    2. Use "Key Files" and "Architecture" sections to identify exactly where to look
    3. Go DIRECTLY to relevant files - don't search broadly
    4. Minimize tool calls - the guide tells you where things are

    TASK: [task description]

    DELIVERABLES:
    - Answer with specific file paths and line numbers
    - Confidence: HIGH/MEDIUM/LOW
    - METRICS: Read=[n] Grep=[n] Glob=[n] Total=[n] | Files: [list]

Task 2 (WITHOUT_GUIDE):
- subagent_type: "general-purpose"
- run_in_background: false  # IMPORTANT: foreground for token tracking
- prompt: |






After both agents complete, fill in this comparison table:

| Metric | WITHGUIDE |
|--------|------------|---------------|
| Tool calls | | |
| Read | | |
| Grep | | |
| Glob | | |
| Confidence | | |
| Correct? | | |
| Tokens (if available) | | |


\\`bash
.claude/hooks/analyze-benchmark.sh
\\`

Note: The analysis script captures main agent tool calls. Subagent metrics come from self-reporting.

---




 ⭐
[Create task based on project's config patterns]


 ⭐
[Create task to find a specific class/function]


 ⭐⭐
[Create task tracing request through system]


 ⭐⭐⭐
[Create task adding feature following existing patterns]


 ⭐⭐⭐
[Create task with bug symptoms to diagnose]


 ⭐⭐⭐⭐
[Create task with race condition or performance issue]


 ⭐⭐⭐⭐⭐
[Create cross-cutting feature design task]

---



| Criteria | 0 | 1 | 2 | 3 |
|----------|---|---|---|---|
| Completion | Failed | Partial | Minor issues | Correct |
| Efficiency | >50 calls | 30-50 | 15-30 | <15 |
| Quality | Wrong patterns | Partial | Minor deviations | Matches style |

Important: Each task must be specific to the actual codebase. Use information from:

• codebase.json stats (files, functions, classes)
• codebase.json modules (real file paths)
• codebase.json call_graph (real function relationships)
• CODEBASE.md architecture section

---

Step 6B: Infrastructure Benchmark Template

Use this template when repo_type is "terraform", "kubernetes", "ansible", or "cloudformation".

Create .claude/BENCHMARK_INFRASTRUCTURE.md with 7 tasks tailored to IaC navigation.

Infrastructure Task Categories:

1. TASK-1 (⭐ Fundamental): Find a resource by type
2. TASK-2 (⭐ Fundamental): Locate a variable definition
3. TASK-3 (⭐⭐ Intermediate): Trace resource dependencies
4. TASK-4 (⭐⭐⭐ Advanced): Assess blast radius of a change
5. TASK-5 (⭐⭐⭐ Advanced): Add a new resource following patterns
6. TASK-6 (⭐⭐⭐⭐ Expert): Debug a dependency cycle
7. TASK-7 (⭐⭐⭐⭐⭐ Expert): Design module refactoring

Template:

# {Project Name} Infrastructure Benchmark

**Purpose:** Measure CODEBASE.md effectiveness on infrastructure navigation tasks.

---

## Orchestrator Instructions

You are running a benchmark to compare agent efficiency WITH vs WITHOUT the CODEBASE.md guide.

**IMPORTANT RULES:**
1. Run agents in FOREGROUND (do NOT use run_in_background)
2. Run both agents IN PARALLEL for the same task (single message, two Task calls)
3. Wait for both to complete before reporting results
4. Report token usage from the Task tool response if available
5. Track metrics from agent self-reports

---

## How to Run

### 1. Reset Logs
\`\`\`bash
.claude/hooks/reset-benchmark.sh
\`\`\`

### 2. Run Both Agents in Parallel (Foreground)

For each task, spawn BOTH subagents in a SINGLE message using two Task tool calls.
Do NOT use run_in_background - we need the full response including token usage.

\`\`\`
Task 1 (WITH_GUIDE):
- subagent_type: "general-purpose"
- run_in_background: false  # IMPORTANT: foreground for token tracking
- prompt: |
    AGENT_WITH_GUIDE: BENCHMARK

    You have access to .claude/CODEBASE.md - an infrastructure navigation guide.

    WORKFLOW:
    1. Read .claude/CODEBASE.md FIRST
    2. Use "Resource Inventory" and "Blast Radius" sections to identify targets
    3. Go DIRECTLY to relevant .tf files - don't search broadly
    4. Minimize tool calls - the guide tells you where things are

    TASK: [task description]

    DELIVERABLES:
    - Answer with specific file paths and line numbers
    - List affected resources if applicable
    - Confidence: HIGH/MEDIUM/LOW
    - METRICS: Read=[n] Grep=[n] Glob=[n] Total=[n] | Files: [list]

Task 2 (WITHOUT_GUIDE):
- subagent_type: "general-purpose"
- run_in_background: false  # IMPORTANT: foreground for token tracking
 prompt: |






After both agents complete, fill in this comparison table:

| Metric | WITHGUIDE |
|--------|------------|---------------|
| Tool calls | | |
| Read | | |
| Grep | | |
| Glob | | |
| Confidence | | |
| Correct? | | |
| Tokens (if available) | | |


\\`bash
.claude/hooks/analyze-benchmark.sh
\\`

---




 ⭐
 Find all resources of type  (e.g., awsgroup, awsrole).
 List all instances with file paths and line numbers.
[Use resources from codebase.json stats.providers to pick a common type]


 ⭐
 Find where variable  is defined and list all resources that use it.
 Variable definition location + list of usages.
[Pick a variable with usedgraph]


 ⭐⭐⭐
 If we modify , what resources would be affected?
 Complete blast radius analysis with severity assessment.
[Pick from blasttype}` resource following the existing naming and tagging patterns.
 Code snippet following project conventions.
[Analyze existing resources to determine patterns]


 ⭐⭐⭐⭐
 Resource  fails to create with "dependency not ready" error. Find the root cause.
 Identify missing explicit dependsgraph]


 ⭐⭐⭐⭐⭐
 Refactor  into a reusable module. What inputs/outputs are needed?
 Module interface design with variable list and output values.
[Pick related resources that could be modularized]

---



| Criteria | 0 | 1 | 2 | 3 |
|----------|---|---|---|---|
| Completion | Failed | Partial | Minor issues | Correct |
| Efficiency | >50 calls | 30-50 | 15-30 | <15 |
| Blast Radius | Missed deps | Partial | Minor misses | Complete |

Important: Each task must be specific to the actual infrastructure. Use information from:

• codebase.json stats (resources, modules, providers)
• codebase.json resources (real resource types and names)
• codebase.json dependency_graph (real dependency chains)
• codebase.json blast_radius (high-risk resources)
• CODEBASE.md infrastructure sections

---

Step 7: Report Completion

Tell the user (adjust based on repo type):

For Application Repos:

Benchmark setup complete!

Repo Type: Application Code

Created:

• .claude/hooks/ (5 scripts)
• .claude/settings.json (hook configuration with PreToolUse + PostToolUse)
• .claude/BENCHMARK_ENGINEERING.md (7 tasks)

To run the benchmark:

1. Reset: .claude/hooks/reset-benchmark.sh
2. For each task, spawn two parallel subagents (see BENCHMARK_ENGINEERING.md)
3. Analyze: .claude/hooks/analyze-benchmark.sh

The hooks will:

• Block AGENT_WITHOUT_GUIDE from reading CODEBASE.md
• Log violations when WITHOUT_GUIDE attempts to access restricted files
• Log all tool calls for comparison
• Track agent turns (API round-trips)

Metrics captured:

• Tool call counts per agent
• Tool breakdown (Read, Grep, Glob, etc.)
• Violation attempts (blocked reads)
• Agent turns (completed operations)
• Files accessed by each agent

Expected result: 40-60% fewer tool calls with CODEBASE.md

For Infrastructure/Terraform Repos:

Benchmark setup complete!

Repo Type: Infrastructure (Terraform/IaC)

Created:

• .claude/hooks/ (5 scripts)
• .claude/settings.json (hook configuration with PreToolUse + PostToolUse)
• .claude/BENCHMARK_INFRASTRUCTURE.md (7 infrastructure tasks)

To run the benchmark:

1. Reset: .claude/hooks/reset-benchmark.sh
2. For each task, spawn two parallel subagents (see BENCHMARK_INFRASTRUCTURE.md)
3. Analyze: .claude/hooks/analyze-benchmark.sh

Infrastructure-specific metrics:

• Resource navigation efficiency
• Blast radius identification accuracy
• Dependency tracing completeness
• Variable usage mapping

The hooks will:

• Block AGENT_WITHOUT_GUIDE from reading CODEBASE.md
• Log all .tf file accesses for comparison
• Track how efficiently each agent finds resources/modules

Expected result: 40-60% fewer tool calls with CODEBASE.md, better blast radius accuracy

Notes

• Tasks must reference real files and patterns from this specific codebase
• Do not create generic tasks - analyze codebase.json to find actual components
• Each task should have a clear expected answer based on the codebase
• The benchmark measures navigation efficiency, not code quality
• For IaC repos: Focus on resource discovery, dependency tracing, and blast radius assessment

Technical Limitations

Hooks don't run inside subagents: Claude Code hooks only execute in the main conversation context. Subagent tool calls are not captured by PreToolUse/PostToolUse hooks.

Workaround: Agents self-report their metrics at the end of each task. The prompt includes a METRICS template they must fill out.

What the hooks DO capture:

• Main agent spawning subagents (Task tool calls)
• Violations if WITHOUT_GUIDE tries to read .claude/ files (blocked by hook)

Token usage: Not directly available via hooks. To estimate:

• Tool calls correlate with input/output tokens
• Check API billing dashboard for actual usage
• Or parse conversation transcripts in ~/.claude/projects/