score-trajectory-analysis

name	score-trajectory-analysis
description	Collect historical scores, fit saturation curves, detect inflection points
execution	tactic
used-by	benchmark-archaeology

Score Trajectory Analysis Tactic

Collect historical SOTA scores for a benchmark, arrange as time-series, fit saturation curves, and detect inflection points indicating phase transitions in benchmark difficulty.

Stages

Stage 1: Multi-Source Score Collection

Gather historical scores from multiple sources to build comprehensive timeline.

Sources (search in order):

1. Papers With Code leaderboards (primary)
2. Official benchmark leaderboards/websites
3. Individual papers reporting SOTA (via dare-ss, dare-scholar)
4. Blog posts and technical reports (via brave-search, dare-web)

Per data point, collect:

• Model name and family
• Score (primary metric)
• Date (publication/release date)
• Paper/source reference
• Model size (parameters) if available
• Training data scale if available

Minimum: 10 data points spanning at least 2 years.

Stage 2: Time-Series Arrangement

• Sort by date
• Compute SOTA envelope (monotonically non-decreasing maximum)
• Identify score jumps > 2 standard deviations
• Note human baseline and theoretical ceiling positions
• Flag suspicious entries (unreplicated claims, withdrawn papers)

Stage 3: Curve Fitting

Fit multiple saturation models to the SOTA envelope:

• Logistic: S(t) = L / (1 + e^(-k(t-t0)))
• Exponential decay to ceiling: S(t) = C - (C-S0) * e^(-lambda*t)
• Linear (for pre-saturation benchmarks)
• Piecewise (for benchmarks with phase transitions)

Report goodness-of-fit (R-squared) for each model. Select best-fit.

Stage 4: Saturation/Inflection Detection

Classify benchmark status:

• Pre-saturation: Linear or early logistic, >20% headroom remaining
• Approaching: Mid-logistic, 5-20% headroom, decelerating gains
• Saturated: <5% headroom, gains < noise level
• Supersaturated: Multiple models at ceiling, benchmark no longer discriminative

Detect inflection points:

• Acceleration phases (new paradigm unlocks rapid progress)
• Deceleration phases (diminishing returns begin)
• Step functions (single breakthrough causes discontinuous jump)

Output

trajectory:
  benchmark: string
  metric: string
  data_points: int
  time_span: string
  sota_envelope:
    - {date, score, model, source}
  best_fit_model: logistic|exponential|linear|piecewise
  fit_r_squared: float
  saturation_status: pre-saturation|approaching|saturated|supersaturated
  headroom: float
  inflection_points:
    - {date, type: acceleration|deceleration|step, cause: string}
  estimated_ceiling: float
  time_to_ceiling: string

Yield Report

Metric	Minimum
Data points collected	10
Sources consulted	3
Curve models fitted	3
Saturation classification produced	1