condition-normalization

name	condition-normalization
description	Compare and standardize experimental conditions across papers
execution	tactic
dependencies	{"sops":["compute-normalization","condition-cataloging","performance-table-assembly"]}

Condition Normalization

Purpose

Build a structured understanding of how experimental conditions vary across papers, then define normalization schemes that enable fair method-to-method comparison. Addresses the fundamental problem that papers evaluate under different settings, making raw score comparison misleading.

Stages

Stage 1: Condition Extraction

For each method-paper pair, extract all evaluation conditions:

• Training data: size, version, preprocessing, augmentation
• Hardware: GPU type, memory, training time
• Hyperparameters: learning rate schedule, batch size, epochs
• Evaluation protocol: data split version, ensembling, post-processing
• Random seeds: number of runs, seed selection, variance reported

Yield: Condition vectors per method-paper pair.

Stage 2: Difference Matrix

Build a matrix showing which conditions differ across methods:

• Identify dimensions with high variance across papers
• Identify dimensions that are controlled (same across all)
• Quantify the impact of each dimension on reported scores (if literature exists)

Yield: Condition difference matrix with impact annotations.

Stage 3: Normalization Scheme

Define rules for adjusting scores to common conditions:

• Compute-normalized comparison (score per FLOP)
• Data-normalized comparison (score per training example)
• Time-normalized comparison (score per GPU-hour)
• Identify which normalizations are valid vs. speculative

Yield: Normalization rule set with validity bounds.

Stage 4: Fair Comparison Baseline

Apply normalization to produce fair comparison subsets:

• Controlled comparison: methods evaluated under identical conditions
• Adjusted comparison: methods with score adjustments applied
• Pareto frontier: compute-vs-performance optimal set

Yield: Fair comparison tables with methodology notes.

Minimum Yield

Metric	Floor
Condition dimensions cataloged	5
Methods with full condition vectors	10
Normalization rules defined	3
Fair comparison sets produced	2

SOPs Used

• condition-cataloging (for Stage 1)
• compute-normalization (for Stage 3)
• performance-table-assembly (for Stage 4)

Available SOPs

Optional, no fixed order; the final leaf is always a sop.

SOP	When to use
compute-normalization	Normalize results by compute budget (Pareto analysis)
condition-cataloging	Record evaluation conditions (data splits, hyperparams, hardware, seeds) from a paper
performance-table-assembly	Assemble unified comparison table with confidence interval annotations