benchmark-framework

# Cognitive Skills Benchmarking Framework ## Overview A rigorous framework for A/B/C testing reasoning patterns to empirically determine which cognitive methodologies perform best across problem categories. This framework enables data-driven pattern selection rather than heuristic-based choices. ## Why Benchmark? Different reasoning patterns (ToT, BoT, SRC, HE, AR, DR, AT, RTR, NDF) claim different strengths, but without empirical measurement: - We cannot validate these claims - We cannot quantify trade-offs (quality vs. cost vs. time) - We cannot track improvement over time - Pattern selection remains subjective This framework provides scientific rigor to cognitive skill evaluation. --- ## Benchmark Structure ### Problem Set Organization ``` benchmark-problems/ ├── optimization/ # ToT territory │ ├── easy/ # 5-10 min problems │ ├── medium/ # 15-30 min problems │ └── hard/ # 30-60 min problems ├── exploration/ # BoT territory │ ├── easy/ │ ├── medium/ │ └── hard/ ├── diagnosis/ # HE territory │ ├── easy/ │ ├── medium/ │ └── hard/ ├── security/ # AR territory │ ├── easy/ │ ├── medium/ │ └── hard/ ├── tradeoffs/ # DR territory │ ├── easy/ │ ├── medium/ │ └── hard/ ├── novel/ # AT territory │ ├── easy/ │ ├── medium/ │ └── hard/ ├── time-critical/ # RTR territory │ ├── easy/ │ ├── medium/ │ └── hard/ └── stakeholder/ # NDF territory ├── easy/ ├── medium/ └── hard/ ``` ### Problem Definition Schema ```yaml # problem-template.yaml problem_id: "OPT-001" domain: "optimization" difficulty: "medium" title: "API Rate Limiter Design" description: | Design a rate limiting system for a public API that handles 10,000 requests/second with fair distribution across users. context: constraints: - "Must handle burst traffic gracefully" - "Sub-millisecond latency requirement" - "Distributed deployment across 5 regions" resources: - "Redis cluster available" - "Current architecture uses nginx" evaluation_criteria: - criterion: "Scalability" weight: 0.3 rubric: | 5: Handles 10x traffic with linear cost 4: Handles 5x traffic efficiently 3: Handles 2x traffic 2: Handles current load only 1: Cannot meet requirements - criterion: "Fairness" weight: 0.25 rubric: | 5: Per-user fairness with adaptive limits 4: Per-user fairness with fixed limits 3: Global fairness only 2: Basic fairness, exploitable 1: No fairness consideration - criterion: "Implementability" weight: 0.25 rubric: | 5: Clear implementation path, <1 week 4: Implementation path, 1-2 weeks 3: Requires some research, 2-4 weeks 2: Significant unknowns 1: Impractical to implement - criterion: "Operational Simplicity" weight: 0.2 rubric: | 5: Self-healing, minimal ops burden 4: Standard monitoring/alerting sufficient 3: Requires dedicated monitoring 2: High operational complexity 1: Operational nightmare ground_truth: known_good_solutions: - "Token bucket with Redis MULTI/EXEC" - "Sliding window log with sorted sets" common_pitfalls: - "Race conditions in distributed counting" - "Memory explosion with naive approaches" expert_rating: 4.2 # If available tags: - "distributed-systems" - "performance" - "redis" ``` --- ## Metrics Framework ### Core Metrics | Metric | Type | Range | Description | |--------|------|-------|-------------| | **Quality Score** | Aggregate | 0-100 | Weighted sum of evaluation criteria | | **Confidence** | Self-reported | 0-100% | Pattern's reported confidence in solution | | **Token Cost** | Integer | 0-∞ | Total tokens consumed (input + output) | | **Execution Time** | Duration | ms | Wall-clock time to solution | | **Correctness** | Binary/Partial | 0-1 | Does solution actually work? | | **Completeness** | Percentage | 0-100% | How much of the problem addressed? | | **Human Preference** | Rank | 1-N | Human ranking among alternatives | ### Quality Score Calculation ```python def calculate_quality_score(solution, criteria): """ Calculate weighted quality score from rubric evaluations. Args: solution: The solution being evaluated criteria: List of (criterion, weight, score) tuples Returns: Quality score 0-100 """ weighted_sum = 0 total_weight = 0 for criterion, weight, score in criteria: # Score is 1-5, normalize to 0-20, then weight normalized = (score - 1) * 25 # 1->0, 5->100 weighted_sum += normalized * weight total_weight += weight return weighted_sum / total_weight if total_weight > 0 else 0 ``` ### Efficiency Metrics ```python @dataclass class EfficiencyMetrics: tokens_per_quality_point: float # Lower is better time_per_quality_point: float # Lower is better quality_per_minute: float # Higher is better @classmethod def calculate(cls, quality: float, tokens: int, time_ms: int): return cls( tokens_per_quality_point=tokens / max(quality, 1), time_per_quality_point=time_ms / max(quality, 1), quality_per_minute=(quality * 60000) / max(time_ms, 1) ) ``` ### Confidence Calibration Track how well self-reported confidence predicts actual quality: ```python def calibration_score(predictions: List[Tuple[float, float]]) -> float: """ Calculate calibration: does confidence predict quality? Args: predictions: List of (confidence, actual_quality) pairs Returns: Calibration score (-1 to 1, 1 is perfect) """ if len(predictions) < 10: return None # Insufficient data # Bin by confidence and compare to actual bins = defaultdict(list) for conf, quality in predictions: bin_key = int(conf // 10) * 10 # 0-10, 10-20, etc. bins[bin_key].append(quality) errors = [] for bin_center, qualities in bins.items(): expected = bin_center + 5 # Center of bin actual = sum(qualities) / len(qualities) errors.append(abs(expected - actual)) # Average error, inverted and normalized avg_error = sum(errors) / len(errors) if errors else 50 return 1 - (avg_error / 50) # 0 error -> 1, 50 error -> 0 ``` --- ## A/B/C Testing Protocol ### Experimental Design ```yaml experiment: id: "EXP-2024-001" hypothesis: "ToT outperforms BoT on optimization problems" conditions: - name: "baseline" pattern: "direct_analysis" description: "No specialized pattern" - name: "condition_a" pattern: "tree_of_thoughts" description: "ToT with default parameters" - name: "condition_b" pattern: "breadth_of_thought" description: "BoT with default parameters" - name: "condition_c" pattern: "tree_of_thoughts" parameters: max_branches: 5 pruning_threshold: 0.6 description: "ToT with aggressive pruning" problem_set: domain: "optimization" difficulties: ["medium", "hard"] sample_size: 30 # Problems per condition randomization: seed: 42 counterbalancing: true # Vary problem order controls: temperature: 0.7 # Fixed across conditions max_tokens: 8000 # Fixed across conditions time_limit: 300000 # 5 minutes per problem ``` ### Statistical Requirements #### Sample Size Calculation ```python def required_sample_size( effect_size: float = 0.5, # Cohen's d alpha: float = 0.05, # Significance level power: float = 0.80 # Statistical power ) -> int: """ Calculate minimum sample size for meaningful comparison. For quality score comparisons (continuous 0-100): - Small effect (d=0.2): n=393 per condition - Medium effect (d=0.5): n=64 per condition - Large effect (d=0.8): n=26 per condition """ from scipy import stats # Two-tailed t-test z_alpha = stats.norm.ppf(1 - alpha/2) z_beta = stats.norm.ppf(power) n = 2 * ((z_alpha + z_beta) / effect_size) ** 2 return int(np.ceil(n)) ``` #### Minimum Requirements | Comparison Type | Minimum N | Statistical Test | |-----------------|-----------|------------------| | Two patterns | 30/condition | Independent t-test | | Multiple patterns | 30/condition | ANOVA + Tukey HSD | | Paired (same problem) | 20 problems | Paired t-test | | Win/loss record | 50 comparisons | Sign test | ### Confound Control ```yaml confound_controls: # Problem-level controls problem_randomization: enabled: true seed_per_experiment: true # Order effects counterbalancing: method: "latin_square" wash_out_period: true # Clear context between conditions # LLM variability temperature_control: fixed_temperature: 0.7 multiple_runs: 3 # Run each problem 3x, average # Time effects session_controls: max_problems_per_session: 10 break_between_conditions: true # Evaluator bias blind_evaluation: enabled: true solutions_anonymized: true random_order: true ``` ### Running an A/B/C Test ```python async def run_abc_test(experiment_config: dict) -> ExperimentResults: """ Execute A/B/C test according to protocol. """ results = ExperimentResults(experiment_id=experiment_config['id']) # Load problem set problems = load_problems( domain=experiment_config['problem_set']['domain'], difficulties=experiment_config['problem_set']['difficulties'] ) # Randomize random.seed(experiment_config['randomization']['seed']) random.shuffle(problems) # Sample required number problems = problems[:experiment_config['problem_set']['sample_size']] for problem in problems: problem_results = {} for condition in experiment_config['conditions']: # Clear context (wash-out) await clear_context() # Run pattern start_time = time.time() solution = await run_pattern( pattern=condition['pattern'], parameters=condition.get('parameters', {}), problem=problem, controls=experiment_config['controls'] ) execution_time = time.time() - start_time # Collect metrics problem_results[condition['name']] = { 'solution': solution, 'quality_score': evaluate_quality(solution, problem), 'confidence': solution.confidence, 'tokens': solution.token_count, 'execution_time': execution_time, 'correctness': verify_correctness(solution, problem) } results.add_problem_results(problem.id, problem_results) # Statistical analysis results.analyze() return results ``` --- ## Problem Categories ### 1. Optimization Problems (ToT Territory) **Characteristics**: Single best solution, prunable search space, clear evaluation criteria. ```yaml example_problems: - id: "OPT-001" title: "Database Query Optimization" type: "performance" - id: "OPT-002" title: "Memory Allocation Strategy" type: "resource" - id: "OPT-003" title: "API Response Caching" type: "architecture" expected_pattern_performance: tree_of_thoughts: "primary" breadth_of_thought: "secondary" direct_analysis: "baseline" ``` ### 2. Exploration Problems (BoT Territory) **Characteristics**: Multiple valid solutions, unknown solution space, need diversity. ```yaml example_problems: - id: "EXP-001" title: "Architecture Options for New Service" type: "greenfield" - id: "EXP-002" title: "Possible Causes of Intermittent Bug" type: "diagnostic" - id: "EXP-003" title: "Migration Strategy Alternatives" type: "strategic" expected_pattern_performance: breadth_of_thought: "primary" tree_of_thoughts: "secondary" direct_analysis: "baseline" ``` ### 3. Diagnosis Problems (HE Territory) **Characteristics**: Information uncertainty, need for multiple hypotheses, testing required. ```yaml example_problems: - id: "DIA-001" title: "Production Latency Spike Investigation" type: "performance" - id: "DIA-002" title: "Data Inconsistency Root Cause" type: "data" - id: "DIA-003" title: "Memory Leak Identification" type: "resource" expected_pattern_performance: hypothesis_engine: "primary" self_reflecting_chain: "secondary" direct_analysis: "baseline" ``` ### 4. Security Problems (AR Territory) **Characteristics**: Adversarial thinking, attack vectors, defense in depth. ```yaml example_problems: - id: "SEC-001" title: "Authentication Flow Security Review" type: "authentication" - id: "SEC-002" title: "API Endpoint Vulnerability Assessment" type: "api_security" - id: "SEC-003" title: "Data Encryption Strategy" type: "data_protection" expected_pattern_performance: adversarial_reasoning: "primary" hypothesis_engine: "secondary" direct_analysis: "baseline" ``` ### 5. Trade-off Problems (DR Territory)