benchmark-framework

name	benchmark-framework
description	Rigorous A/B/C testing framework for empirically evaluating reasoning patterns. Use when you need data-driven pattern selection, want to quantify trade-offs between patterns, or need to validate claims about which cognitive methodology performs best. Enables scientific measurement of quality, cost, and time trade-offs across ToT, BoT, SRC, HE, AR, DR, AT, RTR, and NDF patterns.
license	MIT

Cognitive Skills Benchmarking Framework

Overview

A rigorous framework for A/B/C testing reasoning patterns to empirically determine which cognitive methodologies perform best across problem categories. This framework enables data-driven pattern selection rather than heuristic-based choices.

Why Benchmark?

Different reasoning patterns (ToT, BoT, SRC, HE, AR, DR, AT, RTR, NDF) claim different strengths, but without empirical measurement:

• We cannot validate these claims
• We cannot quantify trade-offs (quality vs. cost vs. time)
• We cannot track improvement over time
• Pattern selection remains subjective

This framework provides scientific rigor to cognitive skill evaluation.

---

Benchmark Structure

Problem Set Organization

benchmark-problems/
├── optimization/           # ToT territory
│   ├── easy/              # 5-10 min problems
│   ├── medium/            # 15-30 min problems
│   └── hard/              # 30-60 min problems
├── exploration/           # BoT territory
│   ├── easy/
│   ├── medium/
│   └── hard/
├── diagnosis/             # HE territory
│   ├── easy/
│   ├── medium/
│   └── hard/
├── security/              # AR territory
│   ├── easy/
│   ├── medium/
│   └── hard/
├── tradeoffs/             # DR territory
│   ├── easy/
│   ├── medium/
│   └── hard/
├── novel/                 # AT territory
│   ├── easy/
│   ├── medium/
│   └── hard/
├── time-critical/         # RTR territory
│   ├── easy/
│   ├── medium/
│   └── hard/
└── stakeholder/           # NDF territory
    ├── easy/
    ├── medium/
    └── hard/

Problem Definition Schema

# problem-template.yaml
problem_id: "OPT-001"
domain: "optimization"
difficulty: "medium"
title: "API Rate Limiter Design"
description: |
  Design a rate limiting system for a public API that handles
  10,000 requests/second with fair distribution across users.

context:
  constraints:
    - "Must handle burst traffic gracefully"
    - "Sub-millisecond latency requirement"
    - "Distributed deployment across 5 regions"
  resources:
    - "Redis cluster available"
    - "Current architecture uses nginx"

evaluation_criteria:
  - criterion: "Scalability"
    weight: 0.3
    rubric: |
      5: Handles 10x traffic with linear cost
      4: Handles 5x traffic efficiently
      3: Handles 2x traffic
      2: Handles current load only
      1: Cannot meet requirements

  - criterion: "Fairness"
    weight: 0.25
    rubric: |
      5: Per-user fairness with adaptive limits
      4: Per-user fairness with fixed limits
      3: Global fairness only
      2: Basic fairness, exploitable
      1: No fairness consideration

  - criterion: "Implementability"
    weight: 0.25
    rubric: |
      5: Clear implementation path, <1 week
      4: Implementation path, 1-2 weeks
      3: Requires some research, 2-4 weeks
      2: Significant unknowns
      1: Impractical to implement

  - criterion: "Operational Simplicity"
    weight: 0.2
    rubric: |
      5: Self-healing, minimal ops burden
      4: Standard monitoring/alerting sufficient
      3: Requires dedicated monitoring
      2: High operational complexity
      1: Operational nightmare

ground_truth:
  known_good_solutions:
    - "Token bucket with Redis MULTI/EXEC"
    - "Sliding window log with sorted sets"
  common_pitfalls:
    - "Race conditions in distributed counting"
    - "Memory explosion with naive approaches"
  expert_rating: 4.2  # If available

tags:
  - "distributed-systems"
  - "performance"
  - "redis"

---

Metrics Framework

Core Metrics

Metric	Type	Range	Description
Quality Score	Aggregate	0-100	Weighted sum of evaluation criteria
Confidence	Self-reported	0-100%	Pattern's reported confidence in solution
Token Cost	Integer	0-∞	Total tokens consumed (input + output)
Execution Time	Duration	ms	Wall-clock time to solution
Correctness	Binary/Partial	0-1	Does solution actually work?
Completeness	Percentage	0-100%	How much of the problem addressed?
Human Preference	Rank	1-N	Human ranking among alternatives

Quality Score Calculation

def calculate_quality_score(solution, criteria):
    """
    Calculate weighted quality score from rubric evaluations.

    Args:
        solution: The solution being evaluated
        criteria: List of (criterion, weight, score) tuples

    Returns:
        Quality score 0-100
    """
    weighted_sum = 0
    total_weight = 0

    for criterion, weight, score in criteria:
        # Score is 1-5, normalize to 0-20, then weight
        normalized = (score - 1) * 25  # 1->0, 5->100
        weighted_sum += normalized * weight
        total_weight += weight

    return weighted_sum / total_weight if total_weight > 0 else 0

Efficiency Metrics

@dataclass
class EfficiencyMetrics:
    tokens_per_quality_point: float  # Lower is better
    time_per_quality_point: float    # Lower is better
    quality_per_minute: float        # Higher is better

    @classmethod
    def calculate(cls, quality: float, tokens: int, time_ms: int):
        return cls(
            tokens_per_quality_point=tokens / max(quality, 1),
            time_per_quality_point=time_ms / max(quality, 1),
            quality_per_minute=(quality * 60000) / max(time_ms, 1)
        )

Confidence Calibration

Track how well self-reported confidence predicts actual quality:

def calibration_score(predictions: List[Tuple[float, float]]) -> float:
    """
    Calculate calibration: does confidence predict quality?

    Args:
        predictions: List of (confidence, actual_quality) pairs

    Returns:
        Calibration score (-1 to 1, 1 is perfect)
    """
    if len(predictions) < 10:
        return None  # Insufficient data

    # Bin by confidence and compare to actual
    bins = defaultdict(list)
    for conf, quality in predictions:
        bin_key = int(conf // 10) * 10  # 0-10, 10-20, etc.
        bins[bin_key].append(quality)

    errors = []
    for bin_center, qualities in bins.items():
        expected = bin_center + 5  # Center of bin
        actual = sum(qualities) / len(qualities)
        errors.append(abs(expected - actual))

    # Average error, inverted and normalized
    avg_error = sum(errors) / len(errors) if errors else 50
    return 1 - (avg_error / 50)  # 0 error -> 1, 50 error -> 0

---

A/B/C Testing Protocol

Experimental Design

experiment:
  id: "EXP-2024-001"
  hypothesis: "ToT outperforms BoT on optimization problems"

  conditions:
    - name: "baseline"
      pattern: "direct_analysis"
      description: "No specialized pattern"

    - name: "condition_a"
      pattern: "tree_of_thoughts"
      description: "ToT with default parameters"

    - name: "condition_b"
      pattern: "breadth_of_thought"
      description: "BoT with default parameters"

    - name: "condition_c"
      pattern: "tree_of_thoughts"
      parameters:
        max_branches: 5
        pruning_threshold: 0.6
      description: "ToT with aggressive pruning"

  problem_set:
    domain: "optimization"
    difficulties: ["medium", "hard"]
    sample_size: 30  # Problems per condition

  randomization:
    seed: 42
    counterbalancing: true  # Vary problem order

  controls:
    temperature: 0.7  # Fixed across conditions
    max_tokens: 8000  # Fixed across conditions
    time_limit: 300000  # 5 minutes per problem

Statistical Requirements

Sample Size Calculation

def required_sample_size(
    effect_size: float = 0.5,  # Cohen's d
    alpha: float = 0.05,       # Significance level
    power: float = 0.80        # Statistical power
) -> int:
    """
    Calculate minimum sample size for meaningful comparison.

    For quality score comparisons (continuous 0-100):
    - Small effect (d=0.2): n=393 per condition
    - Medium effect (d=0.5): n=64 per condition
    - Large effect (d=0.8): n=26 per condition
    """
    from scipy import stats

    # Two-tailed t-test
    z_alpha = stats.norm.ppf(1 - alpha/2)
    z_beta = stats.norm.ppf(power)

    n = 2 * ((z_alpha + z_beta) / effect_size) ** 2
    return int(np.ceil(n))

Minimum Requirements

Comparison Type	Minimum N	Statistical Test
Two patterns	30/condition	Independent t-test
Multiple patterns	30/condition	ANOVA + Tukey HSD
Paired (same problem)	20 problems	Paired t-test
Win/loss record	50 comparisons	Sign test

Confound Control

confound_controls:
  # Problem-level controls
  problem_randomization:
    enabled: true
    seed_per_experiment: true

  # Order effects
  counterbalancing:
    method: "latin_square"
    wash_out_period: true  # Clear context between conditions

  # LLM variability
  temperature_control:
    fixed_temperature: 0.7
    multiple_runs: 3  # Run each problem 3x, average

  # Time effects
  session_controls:
    max_problems_per_session: 10
    break_between_conditions: true

  # Evaluator bias
  blind_evaluation:
    enabled: true
    solutions_anonymized: true
    random_order: true

Running an A/B/C Test

async def run_abc_test(experiment_config: dict) -> ExperimentResults:
    """
    Execute A/B/C test according to protocol.
    """
    results = ExperimentResults(experiment_id=experiment_config['id'])

    # Load problem set
    problems = load_problems(
        domain=experiment_config['problem_set']['domain'],
        difficulties=experiment_config['problem_set']['difficulties']
    )

    # Randomize
    random.seed(experiment_config['randomization']['seed'])
    random.shuffle(problems)

    # Sample required number
    problems = problems[:experiment_config['problem_set']['sample_size']]

    for problem in problems:
        problem_results = {}

        for condition in experiment_config['conditions']:
            # Clear context (wash-out)
            await clear_context()

            # Run pattern
            start_time = time.time()
            solution = await run_pattern(
                pattern=condition['pattern'],
                parameters=condition.get('parameters', {}),
                problem=problem,
                controls=experiment_config['controls']
            )
            execution_time = time.time() - start_time

            # Collect metrics
            problem_results[condition['name']] = {
                'solution': solution,
                'quality_score': evaluate_quality(solution, problem),
                'confidence': solution.confidence,
                'tokens': solution.token_count,
                'execution_time': execution_time,
                'correctness': verify_correctness(solution, problem)
            }

        results.add_problem_results(problem.id, problem_results)

    # Statistical analysis
    results.analyze()

    return results

---

Problem Categories

1. Optimization Problems (ToT Territory)

Characteristics: Single best solution, prunable search space, clear evaluation criteria.

example_problems:
  - id: "OPT-001"
    title: "Database Query Optimization"
    type: "performance"

  - id: "OPT-002"
    title: "Memory Allocation Strategy"
    type: "resource"

  - id: "OPT-003"
    title: "API Response Caching"
    type: "architecture"

expected_pattern_performance:
  tree_of_thoughts: "primary"
  breadth_of_thought: "secondary"
  direct_analysis: "baseline"

2. Exploration Problems (BoT Territory)

Characteristics: Multiple valid solutions, unknown solution space, need diversity.

example_problems:
  - id: "EXP-001"
    title: "Architecture Options for New Service"
    type: "greenfield"

  - id: "EXP-002"
    title: "Possible Causes of Intermittent Bug"
    type: "diagnostic"

  - id: "EXP-003"
    title: "Migration Strategy Alternatives"
    type: "strategic"

expected_pattern_performance:
  breadth_of_thought: "primary"
  tree_of_thoughts: "secondary"
  direct_analysis: "baseline"

3. Diagnosis Problems (HE Territory)

Characteristics: Information uncertainty, need for multiple hypotheses, testing required.

example_problems:
  - id: "DIA-001"
    title: "Production Latency Spike Investigation"
    type: "performance"

  - id: "DIA-002"
    title: "Data Inconsistency Root Cause"
    type: "data"

  - id: "DIA-003"
    title: "Memory Leak Identification"
    type: "resource"

expected_pattern_performance:
  hypothesis_engine: "primary"
  self_reflecting_chain: "secondary"
  direct_analysis: "baseline"

4. Security Problems (AR Territory)

Characteristics: Adversarial thinking, attack vectors, defense in depth.

example_problems:
  - id: "SEC-001"
    title: "Authentication Flow Security Review"
    type: "authentication"

  - id: "SEC-002"
    title: "API Endpoint Vulnerability Assessment"
    type: "api_security"

  - id: "SEC-003"
    title: "Data Encryption Strategy"
    type: "data_protection"

expected_pattern_performance:
  adversarial_reasoning: "primary"
  hypothesis_engine: "secondary"
  direct_analysis: "baseline"

5. Trade-off Problems (DR Territory)