Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
Instruções da origem · Visualização somente leitura
name
agentic-harness-engineering
description
Evolve coding agent harnesses automatically using observability-driven iteration with NexAU components
triggers
["How do I set up agentic harness engineering?","Help me evolve a coding agent harness with AHE","Show me how to run harness evolution experiments","Configure AHE for automatic agent improvement","How do I analyze agent traces with AHE?","Run iterative harness optimization with NexAU","Set up E2B templates for AHE experiments","Debug and improve my coding agent with AHE"]
Agentic Harness Engineering (AHE) is an observability-driven system for automatically evolving the harness around a coding agent. The base LLM model remains frozen while AHE iteratively improves the harness components: system prompts, tool descriptions, tool implementations, middleware, skills, sub-agents, and long-term memory.
AHE operates through a three-phase loop:
Evaluate — Run the agent over a dataset, capture full traces
Analyze — Distill traces into root-cause reports using Agent Debugger
# macOS
brew install uv tmux
# Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
sudo apt install -y tmux
Clone and Install
git clone https://github.com/china-qijizhifeng/agentic-harness-engineering.git
cd agentic-harness-engineering
uv sync
Environment Configuration
cp .env.example .env
Required environment variables:
# Main LLM endpoint (used by code_agent and evolve_agent)
LLM_API_KEY="your_api_key_here"
LLM_BASE_URL="https://api.openai.com/v1"# E2B sandbox (required for safe code execution)
E2B_API_KEY="your_e2b_key"# Web search for evolve_agent
SERPER_API_KEY="your_serper_key"
Optional specialized endpoints:
# Stronger model for Agent Debugger
ADB_LLM_API_KEY="your_key"
ADB_LLM_BASE_URL="https://api.anthropic.com/v1"# Specific model for GPT-5.4 experiments
GPT54_LLM_API_KEY="your_key"
GPT54_LLM_BASE_URL="https://api.openai.com/v1"
E2B Sandbox Setup
AHE supports two E2B deployment modes:
SaaS E2B (default):
# Only set E2B_API_KEY, leave E2B_API_URL unset
E2B_API_KEY="your_e2b_key"
# Launch single experiment in tmux background
./scripts/evolve.sh configs/experiments/exp-003-simple-code-gpt54.yaml
# Launch and attach to see logs in real-time
./scripts/evolve.sh --attach configs/experiments/exp-003-simple-code-gpt54.yaml
# Batch launch all experiments
./scripts/evolve.sh --batch
# Resume a paused/crashed experiment
./scripts/evolve-resume.sh runs/exp-003-simple-code-gpt54
Tmux Session Management
# List running experiments
tmux ls# Attach to running experiment
tmux attach -t exp-003-simple-code-gpt54
# Detach from session (keeps running): Ctrl-b d# Kill experiment
tmux kill-session -t exp-003-simple-code-gpt54
Manual Execution Steps
# Run just the evaluation phase
uv run python evolve.py \
--config configs/experiments/exp-003-simple-code-gpt54.yaml \
--run-dir runs/exp-003-simple-code-gpt54 \
--phase evaluate
# Run just the analysis phase
uv run python evolve.py \
--config configs/experiments/exp-003-simple-code-gpt54.yaml \
--run-dir runs/exp-003-simple-code-gpt54 \
--phase analyze
# Run just the improvement phase
uv run python evolve.py \
--config configs/experiments/exp-003-simple-code-gpt54.yaml \
--run-dir runs/exp-003-simple-code-gpt54 \
--phase improve
Configuration
Experiment Configuration Structure
Experiments use a base + overlay pattern:
# configs/base.yaml - shared defaultsdataset:path:"/path/to/harbor-datasets/terminal-bench-2"harbor:max_workers:8timeout:600evolution:max_iterations:10target_pass_rate:0.85# configs/experiments/my-experiment.yaml - overlayextends:../base.yamlexperiment:name:"my-experiment"description:"Testing new middleware configuration"llm:model:"gpt-5.4-turbo"temperature:0.0harbor:max_workers:16# Override base setting
Key Configuration Sections
Dataset Configuration:
dataset:path:"/path/to/harbor-datasets/terminal-bench-2"tasks:# Optional: run subset-"task_001"-"task_002"
# agents/evolve_agent/workspace/skills/analyze_performance.pyfrom nexau.skills import Skill
from typing importDict, AnyclassAnalyzePerformance(Skill):
"""Analyze pass rate trends across iterations."""
name = "analyze_performance"
description = "Analyze performance trends and identify regression patterns"def__init__(self, run_dir: str):
self.run_dir = Path(run_dir)
asyncdefexecute(self, **kwargs) -> Dict[str, Any]:
"""
Analyze pass rates across iterations.
Returns:
trend: "improving", "degrading", or "stable"
current_rate: Current pass rate
best_rate: Best pass rate achieved
recommendations: List of actionable recommendations
"""
iterations = sorted(self.run_dir.glob("iteration_*"))
pass_rates = []
for iter_dir in iterations:
results_file = iter_dir / "evaluation" / "results.json"if results_file.exists():
withopen(results_file) as f:
data = json.load(f)
pass_rates.append(data["pass_rate"])
iflen(pass_rates) < 2:
return {"trend": "insufficient_data"}
# Calculate trend
recent_avg = sum(pass_rates[-3:]) / len(pass_rates[-3:])
earlier_avg = sum(pass_rates[:-3]) / len(pass_rates[:-3])
if recent_avg > earlier_avg + 0.02:
trend = "improving"elif recent_avg < earlier_avg - 0.02:
trend = "degrading"else:
trend = "stable"
recommendations = []
if trend == "degrading":
recommendations.append("Review recent changes for regressions")
recommendations.append("Consider reverting last iteration's edits")
elif trend == "stable":
recommendations.append("Try more aggressive harness modifications")
return {
"trend": trend,
"current_rate": pass_rates[-1],
"best_rate": max(pass_rates),
"recommendations": recommendations
}
Programmatic Trace Analysis
from pathlib import Path
import json
defanalyze_failure_patterns(iteration_dir: Path):
"""Extract common failure patterns from evaluation traces."""
traces_dir = iteration_dir / "evaluation" / "tasks"
failures = []
for task_dir in traces_dir.iterdir():
reward_file = task_dir / "verifier" / "reward.txt"
trace_file = task_dir / "agent" / "nexau_in_memory_tracer.cleaned.json"# Check if task failedif reward_file.exists():
withopen(reward_file) as f:
if f.read().strip() != "1.0": # Failed# Load tracewithopen(trace_file) as tf:
trace = json.load(tf)
failures.append({
"task": task_dir.name,
"steps": len(trace.get("steps", [])),
"last_error": extract_last_error(trace)
})
# Group by error type
error_counts = {}
for failure in failures:
error = failure["last_error"]
error_counts[error] = error_counts.get(error, 0) + 1return {
"total_failures": len(failures),
"error_distribution": error_counts,
"examples": failures[:5]
}
defextract_last_error(trace: dict) -> str:
"""Extract the last error message from a trace."""
steps = trace.get("steps", [])
for step inreversed(steps):
if"error"in step:
return step["error"]
if step.get("tool_result", {}).get("success") isFalse:
return step["tool_result"].get("error", "unknown_error")
return"no_error_found"
Common Patterns
Pattern 1: Iterative Refinement
# Start with a baseline experiment
./scripts/evolve.sh configs/experiments/baseline.yaml
# After 10 iterations, fork the best harness
cp -r runs/baseline/iteration_007/input runs/baseline-v2/iteration_000/input# Run refinement with different configuration
./scripts/evolve.sh configs/experiments/baseline-v2.yaml
Pattern 2: A/B Testing Middleware
# configs/experiments/test-middleware-a.yamlexperiment:name:"test-middleware-a"harness_overrides:middleware:-"context_compaction"-"error_recovery"# configs/experiments/test-middleware-b.yamlexperiment:name:"test-middleware-b"harness_overrides:middleware:-"context_compaction"-"ralph_loop"# Different middleware
Pattern 3: Transfer Learning Across Benchmarks
# Train on Terminal-Bench 2
./scripts/evolve.sh configs/experiments/train-tb2.yaml
# Freeze the best harness
best_iter = "runs/train-tb2/iteration_008"
frozen_harness = "harnesses/tb2-frozen"
cp -r f"{best_iter}/input" frozen_harness
# Test on SWE-bench-Verified (no further evolution)
uv run python evolve.py \
--config configs/experiments/test-swebench.yaml \
--workspace frozen_harness \
--max-iterations 1# Single evaluation, no evolution
Pattern 4: Multi-Model Comparison
models = ["gpt-5.4-turbo", "claude-4-opus", "gpt-5.5-preview"]
for model in models:
config = {
"experiment": {"name": f"compare-{model}"},
"llm": {"model": model},
"extends": "configs/base.yaml"
}
withopen(f"configs/experiments/compare-{model}.yaml", "w") as f:
yaml.dump(config, f)
# Launch experiment
os.system(f"./scripts/evolve.sh configs/experiments/compare-{model}.yaml")
Troubleshooting
E2B Sandbox Issues
Problem: Sandboxes fail to start with "concurrent limit exceeded"
# Check your E2B tier's concurrent sandbox limit# Reduce max_workers in config to stay under the limit
harbor:
max_workers: 4 # Conservative for free tier
Problem: Template build fails with Docker errors
# Ensure Docker credentials are set for private registriesexport DOCKER_REGISTRY_USERNAME="your_username"export DOCKER_REGISTRY_PASSWORD="your_password"# Rebuild specific failed template
uv run python scripts/build_templates.py \
--dataset-dir /path/to/dataset \
--retry-failed \
task_name
Agent Debugger Issues
Problem: Analysis phase produces empty reports
# Check Agent Debugger has access to tracesls runs/my-exp/iteration_001/evaluation/tasks/*/agent/nexau_in_memory_tracer.cleaned.json
# Verify ADB_LLM environment variablesecho$ADB_LLM_API_KEYecho$ADB_LLM_BASE_URL# Run analysis manually with verbose output
uv run python evolve.py \
--config configs/experiments/my-exp.yaml \
--phase analyze \
--verbose
Evolution Stalls
Problem: Evolve Agent makes no changes across iterations
# Check if workspace is read-onlyls -la runs/my-exp/iteration_NNN/evolve/workspace/
# Verify Evolve Agent has proper permissionschmod -R u+w runs/my-exp/iteration_NNN/evolve/workspace/
# Increase evolution temperature for more exploration# In config:
llm:
temperature: 0.3 # Higher = more creative changes
Performance Degradation
Problem: Pass rate decreases after certain iteration
# Identify the regression iterationcat runs/my-exp/iteration_*/evaluation/results.json | grep pass_rate
# Rollback to last good iterationcp -r runs/my-exp/iteration_005/input runs/my-exp/iteration_007/input
# Resume from that point
./scripts/evolve-resume.sh runs/my-exp --from-iteration 7
