terminal-bench-agent-evaluation

name	terminal-bench-agent-evaluation
title	Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.11868
keywords	["agent-benchmark","terminal-tasks","command-line","realistic-evaluation","agent-capability"]
description	Evaluate agents on 89 challenging terminal-based tasks where frontier models score below 65%, providing realistic assessment of command-line interface automation capability. Use when benchmarking agents designed for system administration, automation, or DevOps tasks.

Terminal-Bench: Agent Evaluation on CLI Tasks

This skill provides a comprehensive benchmarking suite for evaluating agents on realistic, hard terminal-based tasks, revealing significant gaps in even frontier models' abilities to automate command-line workflows.

When to Use

• Evaluating CLI automation agents (DevOps, system administration)
• Measuring agent capability on realistic computer-use tasks
• Identifying gaps in agent reasoning for terminal-based workflows
• Benchmarking improvements in agent autonomy and problem-solving
• Comparing agent performance across different models/architectures

When NOT to Use

• Tasks not involving command-line interfaces
• Simple command execution (Terminal-Bench is for complex workflows)
• Systems where safety is critical (tests include risky operations)
• Evaluating GUI-based agents (Terminal-Bench is CLI-specific)

Key Concept

Terminal-Bench 2.0 contains 89 challenging tasks across realistic scenarios:

• File system operations (finding, organizing, searching)
• Process management (monitoring, controlling, debugging)
• Network operations (connectivity, configuration)
• Software installation and management
• Log analysis and troubleshooting

These tasks reflect real-world challenges where even frontier models fail < 65% of the time.

Benchmark Task Categories

Terminal-Bench covers:

• File Management: Complex operations requiring understanding of file types, permissions, search patterns
• System Monitoring: Extracting metrics from multiple sources, identifying patterns
• Configuration: Managing system settings, understanding configuration formats
• Troubleshooting: Diagnosing issues from partial information, using multiple diagnostic tools
• Scripting: Creating bash scripts for complex multi-step workflows

Using the Benchmark

Evaluate an agent on Terminal-Bench tasks:

# Pseudocode for Terminal-Bench evaluation
class TerminalBench:
    def __init__(self, tasks_path="terminal_bench_tasks.json"):
        self.tasks = load_tasks(tasks_path)  

     ():
        results = {: , : , : []}

         task  .tasks:
            :
                
                output = agent.execute_terminal_task(task.description)

                
                 .validate_output(output, task.expected_output):
                    results[] += 
                    results[].append({
                        : task.,
                        : 
                    })
                :
                    results[] += 
                    results[].append({
                        : task.,
                        : ,
                        : 
                    })
             TimeoutError:
                results[] += 

         results