eval-harness

name	eval-harness
description	Systematic evaluation framework for agent capabilities — capability tests, regression suites, failure analysis
version	1.0.0
author	Hermes Cortex
metadata	{"tags":["eval","testing","reliability","autonomous-agents","quality-gates"]}

Eval Harness — Systematic Agent Evaluation

When to Use

• Before deploying new agent behaviors or workflows
• After model upgrades to verify no regression
• Weekly regression testing on holdout test sets
• When debugging recurring failure patterns
• Before promoting experimental features to production

Core Principles

Eval-driven development turns agents from demos into production systems:

1. Define success BEFORE building — what "done" means must be explicit
2. Outcomes > outputs — verify real results, not just generated text
3. Deterministic + model-based grading — use code for objective checks, LLM for nuance
4. Read transcripts — metrics tell WHAT failed, transcripts tell WHY
5. Out-of-sample gating — survivors must pass on unseen data before deployment

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Eval Harness                              │
├─────────────────────────────────────────────────────────────┤
│  1. Task Definitions (YAML/JSON)                            │
│     - Input scenarios (20-50 representative cases)          │
│     - Expected outcomes (verifiable criteria)               │
│     - Grading rubric (deterministic + LLM)                  │
│                                                              │
│  2. Execution Engine                                        │
│     - Runs agent on each task                               │
│     - Captures full trace (Langfuse)                        │
│     - Applies graders                                       │
│     - Aggregates results                                    │
│                                                              │
│  3. Analysis Layer                                          │
│     - Pass/fail metrics                                     │
│     - Failure clustering                                    │
│     - Transcript sampling                                   │
│     - Trend tracking                                        │
└─────────────────────────────────────────────────────────────┘

Eval Types

Capability Evals

Purpose: Measure what the agent CAN do (new feature, new skill)

Characteristics:

• Start low (baseline performance)
• Improve over iterations
• Focus on specific capability

Example: "Agent can deploy a Next.js app behind nginx with SSL"

Regression Evals

Purpose: Ensure agent MAINTAINS learned tasks

Characteristics:

• Should stay near 100% pass rate
• Run on every change
• Include historical failure cases

Example: "Agent still correctly installs cron jobs without duplicates"

Stress Evals

Purpose: Test boundaries and edge cases

Characteristics:

• Ambiguous requirements
• Conflicting patterns in codebase
• Missing context
• Resource constraints

Example: "Agent handles 404 from API gracefully with retry logic"

Workflow

Step 1: Define Eval Tasks

Create evals/<capability-name>.yaml:

name: cron-installation
description: Verify cron jobs are installed correctly without duplicates

tasks:
  - id: fresh-install
    description: Install crons on fresh Hermes setup
    input: "Set up cron jobs for a new Hermes Agent installation"
    expected:
      - 9 cron jobs created
      - No duplicates in jobs.json
      - All scripts exist and are executable
    grading:
      deterministic:
        - cron_count == 9
        - no_duplicates_in_jobs_json
        - all_scripts_executable
      llm_rubric: |
        - Agent verified scripts exist before creating crons
        - Agent checked for existing jobs before adding
        - Agent provided troubleshooting commands

    
          
     
    
         
         
         
    
      
         
         
       
    
        
     
    
          
          
         
    
      
         
         

Step 2: Run Eval Suite

# Run specific eval
python3 ~/.hermes/scripts/run-evals.py --eval cron-installation

# Run all regression evals
python3 ~/.hermes/scripts/run-evals.py --suite regression

# Run with transcript capture
python3 ~/.hermes/scripts/run-evals.py --eval cron-installation --capture-traces

# Run against holdout set (pre-deployment gate)
python3 ~/.hermes/scripts/run-evals.py --eval cron-installation --holdout

Step 3: Analyze Results

Metrics output:

━━━ Eval Results: cron-installation ━━━

Overall: 78% pass (23/30 tasks)

By category:
  Deterministic: 92% (23/25)
  LLM rubric:    60% (18/30)

Failures:
  - existing-install: duplicate detection failed (2/5 runs)
  - force-reinstall: orphaned job cleanup incomplete (3/5 runs)

Transcripts flagged for review:
  - eval-run-20260619-143022-existing-install-trace-7.json
  - eval-run-20260619-143022-force-reinstall-trace-3.json

Step 4: Read Transcripts

Critical practice: Metrics tell you THAT it failed, transcripts tell you WHY.

# Open specific transcript
cat ~/.hermes-cortex/evals/traces/eval-run-20260619-143022-existing-install-trace-7.json | jq

# Or view in Langfuse UI
open https://langfuse.local/project/.../traces/eval-run-20260619-143022-existing-install-trace-7

What to look for:

• Where did the agent go wrong?
• What context was missing?
• Did the agent make assumptions without verifying?
• Did the agent blend conflicting patterns?
• Did the agent stop too early or loop forever?

Step 5: Iterate

Based on transcript analysis:

1. Fix root cause — not just the symptom
2. Update eval — if the eval didn't catch the failure mode
3. Re-run — verify fix + check for regressions
4. Promote to regression — if capability eval passes consistently

Step 6: Gate on Holdout

Before deploying:

# Run on holdout set (unseen test cases)
python3 ~/.hermes/scripts/run-evals.py --eval cron-installation --holdout

# Must pass 90%+ on holdout to deploy

Why holdout matters: Prevents overfitting to known test cases.

Grading Implementation

Deterministic Graders

# evals/graders/cron_graders.py

def cron_count_equals(expected: int, trace: dict) -> bool:
    """Verify expected number of cron jobs created."""
    jobs = trace.get('final_state', {}).get('cron_jobs', [])
    return len(jobs) == expected

def no_duplicates_in_jobs_json(trace: dict) -> bool:
    """Verify no duplicate job names in jobs.json."""
    jobs = trace.get('final_state', {}).get('cron_jobs', [])
    names = [j['name'] for j in jobs]
    return len(names) == len(set(names))

def all_scripts_executable(trace: dict) -> bool:
    """Verify all referenced scripts exist and are executable."""
    scripts = trace.get('final_state', {}).get('scripts', [])
    return all(s['exists'] and s['executable'] for s in scripts)

LLM Rubric Graders

# evals/graders/llm_rubric.py

from langfuse import Langfuse

def grade_with_rubric(trace: dict, rubric: str) -> dict:
    """Use LLM to grade trace against rubric."""
    client = Langfuse()
    
    score = client.score(
        trace_id=trace['id'],
        name="llm-rubric-grade",
        value=0.0,  # Will be updated by LLM
        comment=rubric,
        # Use LLM to evaluate
        source="LLM"
    )
    
    # Run LLM evaluation
    response = client.llm().chat.completions.create(
        model="claude-sonnet-4",
        messages=[
            {"role": "system", "content": "You are an eval grader. Grade the agent trace against the rubric."},
            {"role": "user", "content": f"Rubric:\n{rubric}\n\nTrace:\n{trace['observations']}"}
        ]
    )
    
    # Parse LLM response for score
    # Expected format: {"score": 0.8, "reasoning": "..."}
    
    return {
        "score": parsed_score,
        "reasoning": parsed_reasoning,
        "passed": parsed_score >= 0.7
    }

Failure Analysis

Weekly Failure Report

# Cron job: weekly-failure-analysis (Monday 7am)
python3 ~/.hermes/scripts/analyze-failures.py --week last

Output:

━━━ Weekly Failure Analysis — Week 24, 2026 ━━━

Total failures: 47
Unique failure modes: 8

Top failure modes:
1. Missing context (12 failures)
   - Agent assumed file structure without reading
   - Agent didn't verify script existence before creating crons
   
2. Silent pattern blending (9 failures)
   - Agent mixed class-based and function-based patterns
   - Agent didn't surface conflict, picked one silently
   
3. Premature completion (8 failures)
   - Agent said "done" before verification step
   - Agent skipped regression tests
   
4. Token overflow (7 failures)
   - Context exceeded model limits mid-task
   - Agent lost track of earlier decisions

Recommended fixes:
- Add read-before-write hook to agent-contract skill
- Add conflict surfacing requirement to task contract
- Add checkpoint verification before "complete" status

Full report: ~/.hermes-cortex/evals/reports/weekly-failure-2026-W24.md

Failure Clustering

# Cluster failures by pattern
from sklearn.cluster import KMeans

def cluster_failures(failures: list) -> dict:
    """Group failures by common patterns."""
    # Embed failure traces
    embeddings = [embed_trace(f['trace']) for f in failures]
    
    # Cluster
    kmeans = KMeans(n_clusters=8)
    labels = kmeans.fit_predict(embeddings)
    
    # Group by cluster
    clusters = {}
    for i, label in enumerate(labels):
        clusters.setdefault(label, []).append(failures[i])
    
    # Analyze each cluster for common patterns
    patterns = {}
    for cluster_id, cluster in clusters.items():
        patterns[cluster_id] = {
            "count": len(cluster),
            "common_features": extract_common_features(cluster),
            "sample_traces": cluster[:3],  # Review these
        }
    
    return patterns

Integration Points

Langfuse Integration

All eval runs are traced in Langfuse:

• Trace name: eval-run-<timestamp>-<task-id>
• Tags: eval, <eval-name>, capability|regression|stress
• Scores: Attached to each trace (deterministic + LLM rubric)
• Observations: Full agent trace with tool calls, outputs, decisions

Session State Integration

Eval runs update session state:

## Current Eval Run

**Eval:** cron-installation
**Started:** 2026-06-19 14:30 KST
**Progress:** 23/30 tasks (77%)
**Current task:** force-reinstall
**Status:** in_progress

Cron Integration

The golden regression gate is ALREADY deployed fleet-wide as an orchestrator cron (F-008): orch-daily-regression-gate at 03:15 daily, no_agent, wired to orch-daily-regression-gate.sh → run-evals.py --suite regression --standalone. Silent on pass; report + exit 1 on fail (Telegram alert to Luke). Do NOT create a second daily-regression cron.

# Manual invocation (same as the cron runs)
python3 ~/.hermes-cortex/scripts/run-evals.py --suite regression --standalone

# Add a NEW golden task → edit evals/regression-golden.yaml, add a grader
# function with @grader("name") in run-evals.py, reference it in the task.

Weekly failure analysis (cluster failures, top patterns) is not yet cron-wired — run manually when needed.

Metrics to Track

Metric	Target	Alert Threshold
Capability eval pass rate	Improve over time	<50% after 3 iterations
Regression eval pass rate	≥95%	<90%
Holdout pass rate	≥90%	<80% (block deployment)
Mean time to detect failure	<24 hours	>48 hours
Failure recurrence rate	<10%	>20% (same failure twice)

Anti-Patterns

❌ Vibe-Driven Development

Wrong: "The agent feels better after the change"

Right: "Regression eval pass rate improved from 87% to 94%"

❌ Overfitting to Eval

Wrong: Agent learns to pass specific eval tasks without generalizing

Right: Holdout set catches overfitting, force generalization

❌ Skipping Transcript Review

Wrong: "All metrics look good, ship it"

Right: "Metrics show 92% pass, but reviewing 5 failed traces reveals a pattern we need to fix"

❌ Eval as Afterthought

Wrong: Build feature, then write evals to prove it works

Right: Write evals first, build to pass evals

Files

Path	Purpose
skills/devops/eval-harness/SKILL.md	This skill
ops/scripts/manage/run-evals.py	Eval execution engine — real deterministic graders in the GRADERS registry; unknown grader names fail loudly, never simulated
ops/scripts/orch-daily-regression-gate.sh	F-008 daily golden gate wrapper (no_agent cron, silent on pass, report+exit 1 on fail)
evals/regression-golden.yaml	Golden task suite v1 — bus send/read, task lifecycle, EXEC round-trip, doctor clean, core skills
evals/suites/regression.yaml	Regression suite manifest (lists regression-golden)
~/.hermes-cortex/evals/traces/	Captured eval traces
~/.hermes-cortex/evals/reports/	Generated reports (JSON, one per run, trend trackable)

Golden suite v1 (F-008, orchestrator daily gate orch-daily-regression-gate at 03:15):

• bus_round_trip — send→read(vt-hidden)→archive probe on inbox_esther via the ACTIVE CORTEX_BUS_URL
• task_lifecycle — task-db.py add→list→start→complete→delete probe (v005 matrix: pending→completed is illegal, so it walks in_progress first; zero residue, safety-net delete in finally)
• exec_round_trip — python3 -c returns EXEC-OK
• doctor_clean — cortex-doctor.py --json summary.fail == 0 (warns tolerated)
• core_skills — every always-section skill in skills.yaml resolves to a loadable SKILL.md

Run: python3 ~/.hermes-cortex/scripts/run-evals.py --suite regression --standalone. Add graders: register a function with @grader("name") returning (passed, detail); reference it in a task's grading.deterministic list.

Related Skills

• change-test-loop — RED-GREEN-REFACTOR for individual tasks
• code-review — Pre-commit review with quality gates
• lesson-aware-agent — Inject lessons from past failures
• auto-remediation — Auto-fix detected issues