knowledge-acquisition

SOTA Performance Baseline Campaign — 5 strategies for systematically collecting, standardizing, and analyzing performance data across methods. Produces standardized comparison tables, progress curves, and headroom analysis.

Assess systematic biases in the evidence body — publication bias, reporting bias, and selective outcome reporting. Budget: 40 studies, 40 effect sizes, 40 web searches.

Track evidence accumulation over time — cumulative meta-analysis protocol design. Budget: 40 studies, 40 effect sizes, 30 web searches.

Design structured data extraction form for systematic meta-analysis data collection

Systematically extract effect sizes and conditions from papers for meta-analytic synthesis

Determine effect size types and calculation methods for meta-analytic synthesis

Build evidence network graph for network meta-analysis — nodes, edges, geometry assessment

Plan the statistical synthesis approach — model selection, heterogeneity strategy, and reporting

Explain why different studies reach different conclusions — heterogeneity investigation protocol. Budget: 30 studies, 30 effect sizes, 50 web searches.

Identify and classify sources of between-study heterogeneity (clinical, methodological, statistical)

Define inclusion/exclusion criteria for systematic study selection in meta-analysis

Cross-Study Statistical Synthesis Campaign — 5 strategies for systematic collection and methodological planning of multi-study evidence synthesis. Covers pairwise, network, cumulative meta-analysis, heterogeneity investigation, and bias detection. Stops at protocol design (no computation).

Produce final meta-analysis protocol document assembling all planning outputs into PRISMA-compliant protocol

Compare N methods simultaneously including indirect evidence — network meta-analysis protocol design. Budget: 50 studies, 80 effect sizes, 60 web searches.

Compare two methods across multiple studies — paired meta-analysis protocol design. Budget: 30 studies, 30 effect sizes, 40 web searches.

Construct PICO/PECO framework for the meta-analysis research question

Plan funnel plots, Egger's test, trim-and-fill, p-curve, and selection model analyses for publication bias

Methodological quality and bias risk assessment of included studies using validated tools

Assess methodological bias using RoB2, PROBAST, or QUADAS-2 validated tools

Design leave-one-out, influence diagnostics, subgroup analyses, and robustness checks

Detect annotation artifacts and shortcuts in benchmarks

Evaluation Methodology Archaeology Campaign — 5 strategies for systematic analysis of AI/ML benchmarks, metrics, and leaderboards. Reveals construct validity issues, saturation, data contamination, and evaluation protocol inconsistencies.

Systematic quality assessment using BetterBench 46-criterion framework — 5 benchmarks, 30 papers, 40 web searches

Identify and catalog all relevant benchmarks in target domain

Produce final structured audit report

Build capability taxonomy, map existing benchmark coverage

Evaluate whether benchmark measures its claimed capability

Detect train-test data leakage and memorization artifacts

Map evaluation coverage, identify untested capability dimensions — 20 benchmarks, 30 papers, 50 web searches

Assess documentation completeness against BetterBench/Datasheets standards

Compare implementation differences of same benchmark across papers

Analyze leaderboard score distributions, compression, selective reporting

Decompose composite metrics into constituent signals, analyze polarity and ceiling effects

Extract evaluation protocol parameters from papers

Analyze evaluation protocol differences across papers for same benchmark — 5 benchmarks, 60 papers, 30 web searches

Track score trajectories, detect saturation/failure points — 15 benchmarks, 50 papers, 60 web searches

Collect historical scores, fit saturation curves, detect inflection points

Challenge construct validity — does benchmark measure claimed capability? — 3 benchmarks, 40 papers, 30 web searches

Patent claim syntax parsing — independent/dependent relationships and element extraction

Determine patent legal status — active, expired, pending, lapsed, or revoked