evaluation-protocol-comparison

name	evaluation-protocol-comparison
description	Compare implementation differences of same benchmark across papers
execution	tactic
used-by	benchmark-archaeology

Evaluation Protocol Comparison Tactic

Compare how different papers implement the same benchmark to expose hidden protocol variance that undermines cross-paper score comparability.

Stages

Stage 1: Paper Collection (Same Benchmark)

Collect 10-15 papers that report results on the target benchmark:

• Prioritize diversity: different labs, years, model families
• Include the original benchmark paper as reference protocol
• Include papers from different venues (top conferences, workshops, preprints)
• Search via dare-ss (ss_relevance_search) and dare-scholar (paper_searching)

Search queries: "[benchmark name] evaluation", "[benchmark name] results", "[benchmark name] state-of-the-art"

Stage 2: Protocol Element Extraction

For each paper, run protocol-element-extraction SOP to extract:

Element Category	Specific Parameters
Data	Split version, subset selection, preprocessing, filtering
Prompting	Template format, few-shot examples (count, selection), instruction wording
Generation	Decoding strategy, temperature, top-p/top-k, max tokens, stop criteria
Evaluation	Metric implementation, postprocessing, normalization, scoring script version
Infrastructure	Framework, precision (fp16/bf16/fp32), batch size, hardware

Stage 3: Difference Matrix Construction

Build a comparison matrix:

• Rows = protocol elements
• Columns = papers
• Cells = specific value used
• Highlight deviations from original protocol

Compute per-element variance:

• None: All papers use identical value
• Low: Minor variations (e.g., different random seeds)
• Medium: Substantive differences (e.g., different few-shot examples)
• High: Fundamental disagreements (e.g., different splits, different metrics)
• Extreme: Papers appear to evaluate different things under same name

Stage 4: Impact Assessment

For each high-variance element:

• Search for ablation studies showing impact of that element
• Estimate score range attributable to protocol choice vs model quality
• Identify which protocol choices systematically favor certain model families
• Flag "protocol p-hacking" — suspicious correlation between protocol choice and reported improvement

Output

protocol_comparison:
  benchmark: string
  papers_compared: int
  reference_protocol: string  # original benchmark paper
  difference_matrix:
    - element: string
      category: data|prompting|generation|evaluation|infrastructure
      variance_level: none|low|medium|high|extreme
      values: list[{paper, value}]
      impact_estimate: string
  highest_variance_elements:
    - element: string
      score_impact: string
      favors: string  # which model family benefits
  protocol_p_hacking_flags:
    - paper: string
      suspicious_choice: string
      benefit: string
  cross_paper_comparability: high|moderate|low|unreliable
  standardization_recommendations:
    - element: string
      recommended_value: string
      rationale: string

Yield Report

Metric	Minimum
Papers compared	8
Protocol elements extracted per paper	10
High-variance elements identified	2
Impact estimates produced	3