exp-bench-ai-research

name	exp-bench-ai-research
title	EXP-Bench: Can AI Conduct AI Research Experiments?
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2505.24878
keywords	["AI Research","Benchmarking","Automation","Multi-Step Reasoning","Experimental Design"]
description	Evaluate AI systems' ability to conduct autonomous research experiments using EXP-Bench, a benchmark for multi-step scientific reasoning and iterative experimental workflows.

Benchmark AI Capability to Conduct Autonomous Experiments

EXP-Bench addresses a critical capability gap: while AI systems excel at isolated tasks, they struggle with multi-step experimental workflows—the core of scientific research. The benchmark evaluates whether AI can design experiments, run them, analyze results, and iterate based on findings. This requires reasoning about experimental design, error handling, result interpretation, and iterative refinement.

The key insight is that research experiments are complex workflows involving multiple decision points: What parameters to test? How to interpret unexpected results? When to pivot to a different approach? EXP-Bench measures these higher-order research capabilities beyond single-task performance.

Core Concept

EXP-Bench evaluates AI research capability through:

• Experimental design: Choosing appropriate parameters, baselines, and evaluation metrics
• Workflow execution: Running multi-step experimental pipelines reliably
• Result analysis: Interpreting experimental outputs and drawing conclusions
• Iterative refinement: Adjusting experiments based on results
• Error handling: Recovering from failures and debugging
• Documentation: Recording experimental setup, results, and conclusions

Success requires reasoning about causal relationships, hypothesis testing, and scientific methodology—not just task completion.

Architecture Overview

• Experiment specification language: Format for describing experiments (parameters, steps, evaluation)
• Execution engine: Runs experimental pipelines with error recovery
• Analysis module: Interprets results and identifies patterns
• Decision maker: Decides on next steps (continue, refine, pivot)
• Logging system: Records all experimental details for reproducibility
• Multi-step reasoning: Chains decisions across multiple experimental phases
• Error detection: Identifies when experiments fail or results are anomalous

Implementation

Build a framework for autonomous experiment execution and analysis:

# EXP-Bench: Autonomous research experiment capability
import json
import subprocess
from dataclasses import dataclass
from typing import List, Dict, Any, Optional
import numpy as np

@dataclass
class ExperimentStep:
    """Specification for one step in an experiment"""
    name: str
    command: str  # Shell command or Python code to execute
    parameters: Dict[str, Any]  # Configuration parameters
    expected_output: str  # What we expect to capture
    error_handling: str  # How to handle failures

@dataclass
class ExperimentResult:
    """Results from executing one experiment"""
    step_name: str
    success: bool
    output: str
    metrics: Dict[str, float]
    error: Optional[str] = None
    timestamp: Optional[str] = None

class AutonomousExperimentRunner:
    """
    Execute multi-step experiments autonomously with iterative refinement.
    """
    def ():
        .max_iterations = max_iterations
        .timeout_per_step = timeout_per_step
        .experiment_history = []

     () -> [ExperimentStep]:
        
        design_prompt = 
        
        
         [
            ExperimentStep(
                name=,
                command=,
                parameters={},
                expected_output=,
                error_handling=
            )
        ]

     () -> [, ]:
        
        results = []
        iteration = 
        success = 

         iteration < .max_iterations   success:
            iteration_results = []

             step  experiment_steps:
                result = ._execute_step(step)
                iteration_results.append(result)

                
                  result.success  step.error_handling == :
                    

                
                analysis = ._analyze_intermediate_results(iteration_results)

                
                 analysis[]:
                    
                    experiment_steps = ._refine_experiment(
                        experiment_steps,
                        analysis[],
                        hypothesis
                    )
                      

            results.extend(iteration_results)

            
            final_analysis = ._analyze_complete_results(results, hypothesis)
            success = final_analysis[]

              success  iteration < .max_iterations - :
                
                refinement = ._generate_refinement(final_analysis, hypothesis)
                experiment_steps = refinement

            iteration += 

         {
            : success,
            : iteration,
            : results,
            : final_analysis,
            : ._generate_conclusions(results, final_analysis)
        }

     () -> ExperimentResult:
        
        :
            
            result = subprocess.run(
                step.command,
                shell=,
                capture_output=,
                timeout=.timeout_per_step,
                text=
            )

            
            output = result.stdout + result.stderr
            success = result.returncode == 

            
            metrics = ._parse_metrics(output, step.expected_output)

             ExperimentResult(
                step_name=step.name,
                success=success,
                output=output,
                metrics=metrics,
                error=result.stderr   success  
            )

         subprocess.TimeoutExpired:
             ExperimentResult(
                step_name=step.name,
                success=,
                output=,
                metrics={},
                error=
            )
         Exception  e:
             ExperimentResult(
                step_name=step.name,
                success=,
                output=,
                metrics={},
                error=(e)
            )

     () -> [, ]:
        
        analysis = {
            : ,
            : [],
            : []
        }

         result  results:
              result.success:
                analysis[].append()

        
         (results) > :
            
            latest_metrics = results[-].metrics
              latest_metrics:
                analysis[].append()
                analysis[] = 

         analysis

     () -> [, ]:
        
        all_metrics = {}
         result  results:
            all_metrics.update(result.metrics)

        analysis = {
            : (r.success  r  results),
            : all_metrics,
            : ,
            : 
        }

        
           all_metrics:
            
            analysis[] = all_metrics[] > 
            analysis[] = all_metrics[]

         analysis

     () -> [ExperimentStep]:
        
        refinement_prompt = 
        
        
         steps

     () -> [, ]:
        
        metrics = {}

        
         re
        patterns = {
            : ,
            : ,
            : ,
            : 
        }

         metric_name, pattern  patterns.items():
             = re.search(pattern, output, re.IGNORECASE)
             :
                metrics[metric_name] = (.group())

         metrics

     () -> [ExperimentStep]:
        
        
         []

     () -> :
        
        conclusion = 
         conclusion

Implement an experimental workflow coordinator:

class ResearchWorkflowCoordinator:
    """
    Coordinate multiple related experiments for iterative research.
    """
    def __init__(self, research_goal: str):
        self.research_goal = research_goal
        self.experiments = []
        self.conclusions = []

    def plan_research_direction(self) -> List[str]:
        """
        Generate sequence of experiments to answer research question.
        """
        planning_prompt = f"""
Research Goal: {self.research_goal}

Plan a sequence of {3-5} experiments to systematically investigate this goal.
Each experiment should build on previous findings.

For each experiment specify:
1. Hypothesis being tested
2. Key variables to manipulate
3. Control conditions
4. Success criteria

Return as JSON array with experiments.
"""
        # In practice, use LLM to plan
        return []

    def run_research_cycle(self) -> Dict[str, Any]:
        """Execute planned sequence of experiments"""
        runner = AutonomousExperimentRunner()
        cycle_results = []

        for exp_spec in self.planned_experiments:
            print(f"\nRunning experiment: {exp_spec['name']}")

            # Design experiment
            steps = runner.design_experiment(exp_spec['hypothesis'])

            
            result = runner.execute_experiment_workflow(steps, exp_spec[])
            cycle_results.append(result)

            
            insights = ._extract_insights(result)

            
              result[]:
                ()
                next_action = ._decide_next_action(result)
                 next_action == :
                    ()
                    

         {
            : .research_goal,
            : (cycle_results),
            : cycle_results,
            : (r[]  r  cycle_results)
        }

     () -> []:
        
        insights = []
         result[]:
            insights.append()
        :
            insights.append()
         insights

     () -> :
        
         result[]:
             
         (.experiments) < :
             
        :
             

Practical Guidance

Aspect	Recommendation	Notes
Max iterations per experiment	3 - 5	More allows refinement; raises cost/time
Step timeout	300 - 600 seconds	Prevents hanging; adjust for your domain
Metric extraction	Regex + structured output	Parse common metrics from output
Error handling	Fail-fast vs retry	Depends on error severity
Experiment documentation	JSON + markdown	Enables reproducibility and analysis

When to use EXP-Bench approach:

• Evaluating AI capability on research tasks
• Building autonomous research systems
• Need multi-step experimental reasoning
• Want to benchmark scientific methodology capability
• Developing AI research assistants

When NOT to use:

• Single-task performance evaluation (use standard benchmarks)
• Experiments don't have verifiable/parseable outputs
• Iterative refinement isn't needed (pre-determined workflow sufficient)
• Computational budget is extremely limited
• Experiments require human judgment for interpretation

Common pitfalls:

• Metrics not properly parsed from outputs (malformed detection)
• Experiments too complex for iterative refinement (define simpler cycles)
• No error recovery mechanism (system fails on first error)
• Iterations don't actually improve results (refinement logic too simple)
• Not tracking experimental history (hard to learn from failures)
• Assuming AI will match human research intuition (it won't, needs supervision)

Reference

EXP-Bench: Can AI Conduct AI Research Experiments? https://arxiv.org/abs/2505.24878