mlsys-experiments

name	mlsys-experiments
description	Use when designing or auditing the evaluation of an MLSys paper, selecting representative workloads and hardware, tuning baselines symmetrically, reporting throughput, latency tails, memory, cost, and quality together, structuring ablations that attribute gains to mechanisms, and building scaling and sensitivity evidence reviewers trust.

MLSys Experiments

Use this before the evaluation is frozen. An MLSys evaluation exists to answer one compound question: does the named mechanism deliver the claimed system-level payoff on workloads that matter, at acceptable cost, for understood reasons? Each clause needs its own evidence, and this venue — which published the MLPerf Training methodology in its own proceedings — holds measurement to benchmark-committee standards.

Workload selection: the make-or-break choice

"Unrepresentative workload" is the modal fatal objection at this venue. Defend the choice explicitly:

• Anchor on recognizable workloads: MLPerf-family tasks, widely used open models, or published request traces. Bespoke workloads need a characterization section showing their statistics (request rates, sequence lengths, arrival burstiness, model mix).
• Cover the axis your mechanism exploits: a scheduler for bursty load must be tested on bursty and uniform load; the uniform case bounds your claim, it does not weaken it.
• Include at least one workload where you expect little or no gain, and show it. A non-win row is the cheapest credibility purchase available.
• State workload licensing/provenance — traces you cannot release constrain your future artifact badges (see mlsys-artifact-evaluation).

Baseline discipline

• Compare against the current strongest systems, version-pinned, in their recommended configurations — and tune them with the same budget you gave your own system. Document both budgets; asymmetric tuning is treated as invalidating.
• Include the "do nothing" configuration where meaningful (e.g., default framework behavior); reviewers use it to sanity-check absolute numbers.
• If a relevant baseline cannot run in your setting, demonstrate the incompatibility rather than asserting it.

The reporting quartet

Never report throughput alone. Each headline claim carries four coordinates:

Coordinate	Report as	Classic gaming pattern it prevents
Throughput/goodput	Under a stated latency constraint	Batch-size inflation that destroys tails
Latency	p50 and p99 (p99.9 for serving), distributions not means	Mean-only tables hiding stragglers
Memory	Peak device + host, at the measured configuration	Wins that only fit on 80GB parts, unstated
Cost/efficiency	$ or joules per unit work, with price/meter basis	"Faster" systems that are 4x more expensive

Quality (accuracy/perplexity or task metric) rides along whenever the technique could plausibly change model output — quantization, sparsity, scheduling with dropping, approximation of any kind. "No quality change" is a claim requiring a measurement.

Ablation = attribution

Systems papers stack techniques; reviews ask which component pays.

• One ablation row per named mechanism, cumulative or leave-one-out — pick one scheme and say which.
• Ablate the insight, not only the code: if the claim is "lookahead scheduling wins," include the no-lookahead scheduler, not just the whole-system-off row.
• Sensitivity sweeps over the parameters a deployer must set (window sizes, thresholds); a mechanism needing per-workload hand-tuning must say so.

Scaling and sensitivity evidence

# experiment-matrix.yaml — one row per plotted point, generated, never hand-edited
sweep:
  models:    [llama-class-8b, llama-class-70b, moe-16e]
  hardware:  [1xA100, 8xA100-nvlink]        # state interconnect explicitly
  load:      [uniform-qps, bursty-trace-A]
  precision: [bf16, int4]
  trials: 5
  warmup_steps: 20
  metrics: [goodput@p99<250ms, p50, p99, peak_mem_gb, gpu_hours, quality]
  seeds: [1, 2, 3, 4, 5]

• Plot trends, not points: gain versus model size, versus load, versus GPU count. A single-configuration win invites the "will this survive scale?" objection you cannot answer in the four-day response window.
• Log everything; regenerate every figure from logs by script (measurement mechanics and variance rules live in mlsys-reproducibility).

Statistical floor for systems numbers

Systems papers rarely need hypothesis-testing machinery, but they always need variance honesty; the floor this venue's reviewers apply:

• Five or more trials for any number in a comparison table; report spread (stdev or IQR) and say which in the caption.
• Overlapping spreads mean the difference is not a finding — either run more trials or report the tie honestly; a "win" inside noise is worse than a reported tie because one reviewer rerun destroys it.
• For latency distributions, trials are not enough: within-run percentiles need enough requests that p99 is estimated from hundreds of samples, not three.
• Seed variation matters only where model quality enters the claim; systems noise (placement, thermal state, co-tenants) dominates latency variance and is controlled by measurement design, not seeds — the split is detailed in mlsys-reproducibility.

Evaluation-design review before running

Run this table exercise before burning GPU-hours: for each claim in the abstract, write the figure that will support it, the workload it needs, and the strongest objection a systems reviewer could raise — then check the matrix covers that objection. Claims without a planned figure get cut from the abstract, not padded later.

Common reject patterns to design against

• Microbenchmark gains that never appear end-to-end; always include the end-to-end run.
• Gains measured only at one scale/hardware and claimed generally.
• Quality silently degraded by the optimization, discovered by a reviewer from your own appendix table.
• Cost omitted where the honest accounting would erase the win.

Output format

[Evaluation readiness] strong / adequate / weak
[Workload defense] <named anchors + characterization + non-win case present?>
[Baseline symmetry] <versions pinned, tuning budgets equal + documented?>
[Quartet coverage] <throughput/latency-tails/memory/cost per headline claim>
[Attribution] <mechanism -> ablation row>
[Decision-critical missing run] <one experiment>