agentbench-dbbench-benchmark

name	agentbench-dbbench-benchmark
description	Run AgentBench DBBench benchmark against LLMs and generate standardized comparison dashboard. Use when testing any model on DBBench (SQL generation task).
version	1.0.0
created	2026-07-21T00:00:00.000Z

AgentBench DBBench Benchmark

Run standardized DBBench benchmark on AgentBench FC and generate comparison dashboard with identical plot style for all models.

Prerequisites

1. AgentBench repo at /tmp/AgentBench (git clone if missing)
2. Docker images already built (agentbench-fc-dbbench-std)
3. 10 dbbench workers running (containers agentbench-fc-dbbench-std-1 through agentbench-fc-dbbench-std-10)
4. LLM endpoint reachable (API URL + model name)

Setup

1. Start DBBench Workers

# Start 10 dbbench workers (320 total capacity)
for i in $(seq 1 10); do
    docker start agentbench-fc-dbbench-std-$i
done

# Verify workers registered
curl -s http://localhost:5020/api/list_workers | python3 -m json.tool
# Should show 'dbbench-std' with 10 workers

2. Verify API Endpoint

# Test API is reachable
curl -s -o /dev/null -w "%{http_code}" \
  -X POST "http://YOUR_HOST:8000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{"model":"test","messages":[{"role":"user","content":"hi"}]}'
# Expected: 401 or 404 (model not found) — NOT connection refused

Running Benchmark

Runner Script Template

Create runner script at /tmp/run_dbbench_N.py:

#!/usr/bin/env python3
"""DBBench benchmark runner - single-turn SQL generation mode."""

import requests
import json
import time
import re

API_URL = "http://YOUR_HOST:8000/v1/chat/completions"  # Replace
MODEL = "your-model-name"                              # Replace
CONTROLLER = "http://localhost:5020"
N = 300                                                 # DBBench has 300 samples total

def call_llm(messages, tools=None):
    payload = {"model": MODEL, "messages": messages, "stream": False,
               "temperature": 0.1, "max_tokens": 4096}
    if tools:
        payload["tools"] = tools
        payload["tool_choice"] = "auto"
    resp = requests.post(API_URL, json=payload,
                         headers={"Authorization": "Bearer EMPTY",
                                      "Content-Type": "application/json"},
                         timeout=180)
    if resp.status_code != 200:
        return None
    msg = resp.json().get("choices", [{}])[0].get("message", {})
    return msg.get("content") or msg.get("reasoning", )

 ():
    
       s:
         m  re.findall(, s, re.DOTALL):
             (k  m.upper()  k  [, , , ,
                                            , , , ]):
                 
     (k  s.upper()  k  [, , , ])

 ():
    results = []
    t0 = time.time()
    sql_n = 
    err_n = 

    ()
    ( * )

     idx  (N):
        t1 = time.time()

        
        r = requests.post(,
                          json={: , : idx}, timeout=)
         r.status_code != :
            results.append({: idx, : ,
                            : r.text[:], : ,
                            : time.time()-t1})
            err_n += 
            

        data = r.json()
        msgs = data.get(, [])
        tools = data.get(, )

        
        resp_text = call_llm(msgs, tools)
         resp_text  :
            results.append({: idx, : , : ,
                            : time.time()-t1})
            err_n += 
            

        has_sql = is_sql(resp_text)
         has_sql:
            sql_n += 

        
        r2 = requests.post(,
                           json={: idx,
                                 : {: resp_text,
                                                    : }},
                           timeout=)

        t_total = time.time() - t1
        tag =   has_sql  
        ()

        results.append({: idx, : ,
                        : has_sql, : ,
                        : (t_total, )})

    total = time.time() - t0

    ( * )
    ()
    ()

    
    out = {: MODEL, : , : N,
           : (total, ), : sql_n,
           : (sql_n/N*, ), : err_n,
           : (total/N, ), : results}
    out_path = 
     (out_path, )  f:
        json.dump(out, f, indent=)
    ()

     out

 __name__ == :
    run_benchmark()

Run Command

python3 /tmp/run_dbbench_N.py 2>&1 | tee /tmp/dbbench_output.log

Expected duration: ~15-25 min for 300 samples (depends on model speed).

Visualization (Exact Same Style)

After benchmark completes, generate the standardized dashboard:

import re, json, numpy as np
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec

# Parse log
lines = open('/tmp/dbbench_output.log').readlines()
times, has_sql = [], []
for line in lines:
    m = re.match(r'\[(\d+)\]\s+(SQL|NONE)\s+([\d.]+)s', line.strip())
    if m:
        times.append(float(m.group(3)))
        has_sql.append(m.group(2) == 'SQL')

n = len(times)
sql_n = sum(has_sql)

# Read JSON for metadata
with open('/tmp/agentbench_dbbench_300.json') as f:
    data = json.load(f)

total_time = data['elapsed_seconds']
avg_time = total_time / n

# Create 8-panel figure (EXACT layout)
fig = plt.figure(figsize=(18, 12))
gs = gridspec.GridSpec(3, 4, figure=fig, hspace=0.35, wspace=0.3,
                       left=0.06, right=0.96, top=0.92, bottom=0.06)

# Panel 1: Time histogram (top, 2 cols)
ax1 = fig.add_subplot(gs[0, :2])
ax1.hist(times, bins=20, color='#4A90D9', edgecolor='white', linewidth=1.2, alpha=)
ax1.axvline(np.mean(times), color=, linestyle=, linewidth=,
            label=)
ax1.axvline(np.median(times), color=, linestyle=, linewidth=,
            label=)
ax1.set_xlabel(, fontsize=, fontweight=)
ax1.set_ylabel(, fontsize=, fontweight=)
ax1.set_title(, fontsize=, fontweight=)
ax1.legend(loc=, fontsize=)
ax1.grid(axis=, alpha=)


ax2 = fig.add_subplot(gs[, ])
bars = ax2.bar([, ], [sql_n, n-sql_n],
               color=[, ], edgecolor=, linewidth=, width=)
 bar, count  (bars, [sql_n, n-sql_n]):
    ax2.text(bar.get_x()+bar.get_width()/, bar.get_height()+,
             (count), ha=, fontsize=, fontweight=)
ax2.set_ylabel(, fontsize=, fontweight=)
ax2.set_title(, fontsize=, fontweight=)
ax2.grid(axis=, alpha=)
ax2.set_ylim(, (sql_n, n-sql_n)*)


ax3 = fig.add_subplot(gs[, :])
colors_sample = [  s    s  has_sql]
ax3.bar((n), times, color=colors_sample, edgecolor=, linewidth=, width=)
ax3.set_xlabel(, fontsize=, fontweight=)
ax3.set_ylabel(, fontsize=, fontweight=)
ax3.set_title(,
              fontsize=, fontweight=)
ax3.set_xticks((, n, ))
ax3.grid(axis=, alpha=)
ax3.axhline(, color=, linestyle=, alpha=, linewidth=, label=)
ax3.legend(fontsize=)


ax4 = fig.add_subplot(gs[, ])
cum_rate = np.cumsum(has_sql) / np.arange(, n+) * 
ax4.plot((, n+), cum_rate, linewidth=, color=)
ax4.axhline(sql_n/n*, color=, linestyle=, linewidth=,
            alpha=, label=)
ax4.set_ylabel(, fontsize=)
ax4.set_title(, fontsize=, fontweight=)
ax4.legend(fontsize=)
ax4.grid(alpha=)
ax4.set_ylim(, )


ax5 = fig.add_subplot(gs[, ])
batch_rates = [(has_sql[i:i+])/*  i  (, n, )]
batches = [  i  (, n, )]
ax5.bar(batches, batch_rates, color=, edgecolor=, linewidth=, width=)
ax5.axhline(sql_n/n*, color=, linestyle=, linewidth=,
            alpha=, label=)
ax5.set_ylabel(, fontsize=, fontweight=)
ax5.set_title(, fontsize=, fontweight=)
ax5.legend(fontsize=)
ax5.grid(axis=, alpha=)
ax5.set_ylim(, )


ax6 = fig.add_subplot(gs[, ])
ax6.axis()
ax6.text(, , , fontsize=, fontweight=)
ax6.text(, , , fontsize=)
ax6.text(, , , fontsize=)
ax6.text(, , , fontsize=,
         fontweight=, color=)
ax6.text(, , , fontsize=)
ax6.text(, , , fontsize=)
ax6.text(, , , fontsize=)
ax6.text(, , , fontsize=)


ax7 = fig.add_subplot(gs[, ])
fast = (  t  times  t < )
med = (  t  times   <= t < )
slow = (  t  times   <= t < )
vslow = (  t  times  t >= )
cats = [, , , ]
counts = [fast, med, slow, vslow]
ax7.bar(cats, counts, color=[, , , ],
        edgecolor=, linewidth=, width=)
 bar, count  (ax7.patches, counts):
    ax7.text(bar.get_x()+bar.get_width()/, bar.get_height()+,
             (count), ha=, fontsize=, fontweight=)
ax7.set_ylabel(, fontsize=, fontweight=)
ax7.set_title(, fontsize=, fontweight=)
ax7.grid(axis=, alpha=)


ax8 = fig.add_subplot(gs[, ])
ax8.axis()
summary_lines = [
    ,
    ,
    ,
    ,
    ,
    ,
    ,
    ,
]
y = 
 line  summary_lines:
    color = 
     line.startswith():
        color =   sql_n/n >=   
    ax8.text(, y, line, fontsize=, va=, fontweight=  line.startswith()  , color=color)
    y -= 


plt.suptitle(,
             fontsize=, fontweight=, y=, color=)

plt.savefig(, dpi=,
            bbox_inches=, facecolor=)
()

API Integration Notes

Common API Endpoints

Endpoint	Model	Notes
http://141.33.165.84:8000/v1	aip-best	Qwen3.6-35B-A3B, reliable
http://localhost:11434/api/chat	Ollama models	Local Ollama
Custom endpoints	Various	Replace API_URL in runner

API Response Handling

Some vLLM endpoints return content: null with reasoning in message.reasoning field. Always handle both:

msg = resp.json().get("choices", [{}])[0].get("message", {})
content = msg.get("content") or msg.get("reasoning", "")
if not content:
    return None

Tool Args Format

Tool call arguments may be JSON strings, not dicts:

func_args = tc.get("function", {}).get("arguments", {})
if isinstance(func_args, str):
    func_args = json.loads(func_args)

Multi-Model Comparison

When comparing multiple models:

1. Run each model separately with identical 300-sample benchmark
2. Save each to /tmp/agentbench_dbbench_<model>.json
3. Generate individual plots with same style
4. Create summary table:

Model	Size	SQL Rate	Avg Time	Errors
llama3.2:3b	1.9GB	100%	3.8s	0
aip-best	-	93%	4.9s	0
qwen3.6:latest (local)	23GB	~3%	28s	many

Infrastructure Issues & Fixes

Workers Not Registered

# Check containers
docker ps --format '{{.Names}}' | grep agentbench-fc-dbbench
# Start all 10
for i in $(seq 1 10); do docker start agentbench-fc-dbbench-std-$i; done
# Verify
curl -s http://localhost:5020/api/list_workers | python3 -m json.tool

Controller Returns "task does not exist"

# Worker not running - start it
docker compose -f /tmp/AgentBench/extra/docker-compose.yml up -d dbbench-std

Worker Capacity Exhaustion

If running many samples, 10 workers (320 capacity) may get overwhelmed. Solution: Ensure all 10 containers are running.

Docker SDK Bug (OS Interaction only, not DBBench)

Not relevant for DBBench — this affects os_interaction task only.

Ollama Initialization (Parallel Runs)

If running parallel benchmarks, Ollama can stall. Run single-threaded or with max_workers=1.

File Outputs

File	Description
/tmp/agentbench_dbbench_300.json	Full results data (one JSON per model)
/tmp/agentbench_dbbench_300_results.png	8-panel dashboard plot
/tmp/dbbench_output.log	Raw terminal log with per-sample output

Key Reminders

• DBBench has 300 samples total (not 50, not 100)
• 0 infrastructure errors expected when workers are running
• 93-100% SQL rate is normal for capable models
• 2-5s/sample is normal for fast models; slower models may be 10-30s
• Single-turn mode only — no multi-round interaction (simpler, faster)
• SQL detection checks both code blocks and inline SQL
• Use consistent plot style for all models to enable direct comparison
• Docker containers stay running between benchmark runs — no need to restart