| name | icde-experiments |
| description | Use when designing or auditing IEEE ICDE experiments for a data-engineering paper: workload realism, tuned baselines, scale curves over single points, throughput and tail-latency with declared variance, mechanism-isolating ablations, cost/loss disclosure, and hardware reporting that satisfies a builder-heavy committee. |
ICDE Experiments
Use this before submission when the evaluation is not yet locked. At ICDE, the experiments are
where a systems paper is won or lost; a builder-heavy committee reads them adversarially.
Experiment audit
- Map each performance claim to a figure, table, or measured crossover — no orphan claims.
- Run tuned baselines, not straw men. Document the tuning budget you gave each competitor;
an untuned baseline is the single most common ICDE reject trigger.
- Prefer curves over points: show behavior across scale factors, data sizes, or contention
levels, not one operating point where you happen to win.
- Report throughput and tail latency with variance — multiple runs, and captions that say
whether bars are standard deviations, confidence intervals, or percentiles. A p99 that hides
behind a median is a caught omission.
- Add ablations that isolate the mechanism: turn off the one idea and show the gain
disappears, so the improvement cannot be attributed to unrelated engineering.
- Disclose the cost of every gain: memory, write amplification, CPU, or read-latency
penalty. A loss map is a credibility signal, not a weakness.
- Report hardware, OS, storage device, software versions, dataset construction, and workload
generators with seeds so the numbers are reconstructable.
What experiments are for at this venue
- ICDE experiments exist to establish and isolate a systems effect, not to top a
leaderboard. One clean ablation proving the mechanism causes the gain outweighs five more
datasets where it merely correlates.
- The strongest design pairs a synthetic generator you can sweep (to find the crossover and
stress the mechanism) with a real workload (to show practical relevance). State where the
synthetic regime matches the real one and where it does not.
- Reviewers check whether the operating regime — data size, concurrency, hardware — is one
where the claimed effect should even appear; a cache-resident benchmark cannot support a
claim about I/O-bound behavior.
Evidence-burden table
| Claim | Matching evidence | Reject pattern avoided |
|---|
| "The problem exists" | Baseline stalls/degrades on the target workload | "Solving a problem nobody has" |
| "Our mechanism causes the gain" | Ablation toggling the one idea | "Gain from unrelated engineering" |
| "It scales" | Curve across scale factors, not a single point | "Single-point win, unknown scaling" |
| "The cost is acceptable" | Disclosed memory/CPU/latency cost table | "Free-lunch claim, hidden trade-off" |
| "It generalizes" | A real workload beside the synthetic sweep | "Only synthetic, no real relevance" |
Vignette: evaluating a write-optimized index
A paper claims higher ingestion by deferring ordering. The matching plan: sweep the
append-to-scan ratio synthetically to locate the crossover, run a real telemetry trace at the
claimed rate to show the baseline stalls where the new index does not, ablate the deferral to
prove it is the cause, and report the read-latency tail and memory cost so the trade is legible
— every panel tied to a claim in the text.
Reporting floor
- Multiple runs and reported variance for every performance figure; name the metric precisely
(throughput, p50/p99 latency, amplification).
- Report the hardware and the compute actually consumed, not vague feasibility language.
Output format
[Experiment readiness] strong / adequate / weak
[Claim -> evidence map] <claim: figure/table/crossover>
[Baseline fairness] <tuning budget disclosed? untuned competitors?>
[Missing evidence] <scale curve / variance / ablation / cost disclosure>
[Reproducibility gaps] <hardware / versions / generators / seeds>
[Decision-critical next run] <one experiment>
