agentbench-benchmarking

name	agentbench-benchmarking
description	Complete guide to running AgentBench FC benchmarks with LLM agents
version	1.0.0
created	2026-07-14T00:00:00.000Z

AgentBench Benchmarking

Complete guide to running AgentBench FC (Function Calling) benchmarks with LLM agents.

Overview

AgentBench FC evaluates LLM agents across multiple environments (dbbench, os_interaction, knowledgegraph, webshop, alfworld) using function calling.

Repository: https://github.com/THUDM/AgentBench

Setup

1. Clone and Install

git clone https://github.com/THUDM/AgentBench.git
cd AgentBench
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

2. Build Docker Images

Required images for each task:

# dbbench
docker pull mysql:8

# os_interaction
docker build -t local-os/default -f ./data/os_interaction/res/dockerfiles/default data/os_interaction/res/dockerfiles
docker build -t local-os/packages -f ./data/os_interaction/res/dockerfiles/packages data/os_interaction/res/dockerfiles
docker build -t local-os/ubuntu -f ./data/os_interaction/res/dockerfiles/ubuntu data/os_interaction/res/dockerfiles

3. Start Stack

docker compose -f extra/docker-compose.yml up -d

This starts:

• AgentRL Controller (port 5020)
• Task workers (dbbench, os_interaction, etc.)
• Redis (container allocation)
• Freebase server (for knowledgegraph)

Warning: webshop requires ~16GB RAM. ALFWorld has known memory leaks.

API Structure

Controller Endpoints

Endpoint	Method	Purpose
/api/list_workers	GET	List available task workers
/api/get_indices?name=TASK	GET	Get available sample indices
/api/start_sample	POST	Start a sample (returns initial prompt, NOT session_id)
/api/interact	POST	Submit agent response
/api/cancel	POST	Cancel session

⚠️ API Quirk

start_sample returns the initial prompt directly:

{
  "messages": [...],
  "tools": [...]
}

NOT a session_id. Session management is handled internally by the worker. Use the index as session_id for interact calls.

Running Benchmarks

Option 1: Direct API Runner

Use the runner script pattern:

import requests

# Get initial prompt
resp = requests.post(
    "http://localhost:5020/api/start_sample",
    json={"name": "dbbench-std", "index": 0}
)
data = resp.json()  # Contains messages, tools

# Call your LLM
result = call_llm(data["messages"], data["tools"])

# Submit response
interact_resp = requests.post(
    "http://localhost:5020/api/interact",
    json={
        "session_id": 0,  # Use index as session_id
        "agent_response": {"content": result, "status": "CONTINUE"}
    }
)

See references/agentbench_runner.py for a complete implementation.

Option 2: Assigner Script

The built-in assigner uses complex YAML configs:

python -m src.assigner --config configs/assignments/my_config.yaml

Config format (see references/assignment_config.yaml):

• definition: task assembly + agent config
• assignments: task -> sample count mapping
• concurrency: parallel worker count
• output: results directory

Task Types

Task	Samples	Description
dbbench-std	300	SQL query generation
os-std	144	Linux shell operations
knowledgegraph-std	-	KG reasoning (requires Freebase)
webshop-std	-	E-commerce navigation (~16GB RAM)
alfworld-std	-	Embodied tasks (memory leak)

LLM Integration

External API Endpoint Pattern

For remote/vLLM endpoints (e.g., http://141.33.165.84:8000/v1/chat/completions):

API_URL = "http://YOUR_HOST:8000/v1/chat/completions"
MODEL_NAME = "aip-best"  # or your model name
API_KEY = "EMPTY"  # or your key

def call_api(messages, tools=None):
    payload = {
        "model": MODEL_NAME,
        "messages": messages,
        "stream": False,
        "temperature": 0.1,
    }
    if tools:
        payload["tools"] = tools
    
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json",
    }
    
    resp = requests.post(API_URL, json=payload, headers=headers, timeout=180)
    return resp.json()

⚠️ API Response Quirk: Some vLLM endpoints return reasoning field instead of content:

content = response_msg.get("content") or response_msg.get("reasoning", "")

⚠️ Tool Args Format: tool_calls[0].function.arguments may be a JSON string, not a dict:

func_args = tc.get("function", {}).get("arguments", {})
if isinstance(func_args, str):
    func_args = json.loads(func_args)
query = func_args.get("query", "") if isinstance(func_args, dict) else ""

Ollama

def call_ollama(messages, tools=None):
    payload = {
        "model": "qwen3.6:latest",
        "messages": messages,
        "stream": False,
    }
    if tools:
        payload["tools"] = tools
    
    resp = requests.post("http://localhost:11434/api/chat", json=payload)
    return resp.json()

Function Calling Format

AgentBench expects tool calls in the response. Ollama Qwen models support this natively:

{
  "message": {
    "content": "Explanation...",
    "tool_calls": [
      {
        "function": {
          "name": "execute_sql",
          "arguments": {"query": "SELECT ..."}
        }
      }
    ]
  }
}

Pitfalls

1. API version mismatch: The controller API changed from documented format. start_sample returns prompt directly, not session_id. Use the sample index as session_id for interact calls.

2. Config import paths: Assignment configs use relative imports. Use absolute paths (/tmp/AgentBench/configs/...) to avoid resolution errors.

3. SQL execution: In dbbench, SQL is executed internally by the worker when detected in code blocks. Don't call /execute_sql separately.

4. Memory leaks: ALFWorld worker leaks memory/disk. Restart after ~50 samples.

5. Webshop RAM: Requires ~16GB. Skip if limited resources.

6. Freebase data: knowledgegraph task needs Freebase database at ./virtuoso_db/virtuoso.db. Download from https://github.com/dki-lab/Freebase-Setup.

7. External API quirks: vLLM endpoints may return reasoning instead of content. Tool args may be JSON strings, not dicts. Handle both cases.

8. OS Interaction task Docker SDK bug: The os-std worker fails with AttributeError: 'int' object has no attribute 'connect' at aiodocker/stream.py:52. This is a Docker SDK compatibility issue, not missing data or configuration. All data (144 samples across 7 datasets) and Docker images are present. Fix: Downgrade Docker SDK inside the worker container: docker exec <container> pip install 'docker==6.1.3' 'aiodocker==0.21.0', or use a direct runner that bypasses the container execution layer (see scripts/os_direct_runner.py).

9. Multi-round loop performance: Full multi-round evaluation with 180s timeouts is extremely slow (~2-3 min/sample). For SQL generation quality tests, use single-round mode (~1s/sample). Full 300-sample benchmark with multi-round: ~18 min. Single-round 100-sample: ~1.5 min.

10. Tool usage rate: Models like aip-best (Qwen3.6) may prefer natural language responses over tool calls (~31% tool usage observed). Tune system prompt to encourage tool usage for full benchmark evaluation.

Results Format

Results are saved as JSON:

{
  "model": "qwen3.6:latest",
  "task": "dbbench-std",
  "elapsed_seconds": 435,
  "results": [
    {
      "index": 0,
      "status": "COMPLETED",
      "final_answer": ["Women +60kg Bronze"],
      "rounds": 2,
      "history_length": 5
    }
  ]
}

References

• references/agentbench_runner.py: Complete runner implementation
• references/assignment_config.yaml: Working config template
• references/api_quirks.md: API behavior notes
• references/agentbench-api-runner.md: External API integration guide with benchmarks
• references/plot-generation-pattern.md: Visualization patterns for results
• references/gwgd_benchmark_findings.md: gwgd endpoint (glm-4.7) benchmark analysis
• references/helmholtz_benchmark_analysis.md: Helmholtz Blablador endpoint analysis (rate limits, output format issues)
• references/dbbench_full_benchmark_results.md: Complete DBBench benchmark results and interpretation
• references/os_interaction_benchmark_results.md: OS Interaction benchmark results and infrastructure fixes
• references/kg_benchmark_results.md: Knowledge Graph benchmark results and tool-calling analysis
• references/dbbench-session-report.md: Complete DBBench session analysis
• references/dbbench-verification-pattern.md: Concurrency issue fix (0% → 98% with single-threaded)
• references/agentbench_full_suite_guide.md: Full 10-task benchmark suite implementation
• references/dbbench_300_run_2026-07-21.md: Verified 300-sample full run (92.7% SQL, 0 errors, 4.89s avg, 24.5 min)
• references/llama32_benchmark_results.md: llama3.2:3b benchmark results (100% success, 5x faster than aip-best)
• references/llama32_full_benchmark_results.md: Complete 6-task suite results (246 samples, 100% success, 1.92s avg)
• references/model_comparison_summary.md: Comprehensive model comparison table (llama3.2:3b vs qwen3.6:latest vs deepseek-r1:70b vs aip-best API)
• references/teuken-7b-benchmark-results.md: Teuken-7B full benchmark (206 samples, 87.4% success, 4.81s/sample) - structured tasks excel (99% SQL), reasoning weak (57%)
• scripts/run_all_benchmarks_llama32.py: Full suite runner for llama3.2:3b (recommended)
• scripts/os_direct_runner.py: Direct runner bypassing Docker SDK bug

35. Long benchmark execution (execute_code timeout): The execute_code tool has a 300s hard timeout — any benchmark expected to take >5min will be killed. Fix: Use terminal(background=True) with output piped through tee to a log file, then poll progress with wc -l and tail. Example: python3 /tmp/bench.py 2>&1 | tee /tmp/bench_output.log. Check progress: wc -l /tmp/bench_output.log && tail -5 /tmp/bench_output.log. Verified with the 300-sample DBBench run (24.5 min).

36. aip-best 300-sample DBBench verified result (2026-07-21): Full 300-sample run completed. 278/300 SQL (92.7%), 0 errors, 4.89s avg, 1.0s min, 32.1s max, 1467.4s total (24.5 min). 22 samples did not generate SQL. Data in references/dbbench_300_run_2026-07-21.md.

"File not found" in config: Use absolute paths for imports.

Docker build fails (visdom): ALFWorld has dependency issues. Skip alfworld-std if not needed.

32. Stopping long-running benchmarks: When benchmarks take too long and the user says "stop all":

    • Kill benchmark scripts: pkill -f 'dbbench' or kill <pid>
    • Docker container cleanup (CRITICAL): The agentbench-fc-* worker containers run as root inside Docker. Host-level kill does NOT stop them. You MUST explicitly stop each container:

      docker stop agentbench-fc-dbbench-std-1 agentbench-fc-dbbench-std-2 ... agentbench-fc-dbbench-std-10
      docker stop agentbench-fc-os_interaction-std-1 ... agentbench-fc-os_interaction-std-10
      docker stop agentbench-fc-knowledgegraph-std-1 ... agentbench-fc-knowledgegraph-std-10
      

    • Quick cleanup pattern:

      # Kill all benchmark scripts
      pkill -f 'dbbench' 2>/dev/null
      pkill -f 'benchmark' 2>/dev/null
      # Stop ALL agentbench Docker containers
      docker stop $(docker ps --format '{{.Names}}' | grep 'agentbench-fc') 2>/dev/null
      # Verify clean
      docker ps --format '{{.Names}}' | grep 'agentbench' | wc -l
      

    • Lesson: Always clean up Docker containers when stopping benchmarks. Leaving them running wastes resources and can cause conflicts on restart.

33. Quick worker restart via docker start (2026-07-21): When images are already built, docker compose up can timeout rebuilding heavy images (webshop is 11GB). If containers previously exited but images exist:

    • Check: docker ps -a --filter 'name=agentbench-fc-dbbench' for exited containers
    • Fast restart: docker start agentbench-fc-dbbench-std-{1..10} — starts all 10 workers in <1s
    • Verify: curl -s http://localhost:5020/api/list_workers — workers register within 5s
    • Only use docker compose up when images are missing or need rebuilding
    • Key lesson: docker start vs docker compose up — the former is instant for pre-built images, the latter may hang on large rebuilds

34. Single-turn DBBench benchmark pattern (2026-07-21): For SQL generation quality tests, single-turn mode is much faster than multi-round:

    • Get initial prompt via start_sample
    • Call LLM with full conversation history from controller

"Not Found" on interact: Check that the task worker is running and the session exists.

404 on /v1/chat/completions: Check endpoint URL. Some vLLM instances use different paths.

Empty messages array (os-std): OS Interaction task has known issues. Use dbbench-std instead.