Use AFTER a Search has completed and BEFORE claiming any speedup or shipping an ACF. Loads the dump_results CSV, extracts top-K candidates (single-objective) or the Pareto front (multi-objective), re-measures each against the no-ACF baseline with 100+ trials on fresh caches, runs Welch's t-test plus Cohen's d, rejects three classic false-positive patterns (lucky-min / higher-variance / multiple-comparisons-of-N), and saves the validated winner as best.acf. Triggers on "validate result", "extract best config", "Welch's t-test", "is my speedup real", "save best ACF", "pareto front", "claim speedup", "ship config".
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
The command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
File Explorer
2 files
Showing SKILL.md
SKILL.md
Source instructions · Read-only preview
name
compileiq-validate-result
description
Use AFTER a Search has completed and BEFORE claiming any speedup or shipping an ACF. Loads the dump_results CSV, extracts top-K candidates (single-objective) or the Pareto front (multi-objective), re-measures each against the no-ACF baseline with 100+ trials on fresh caches, runs Welch's t-test plus Cohen's d, rejects three classic false-positive patterns (lucky-min / higher-variance / multiple-comparisons-of-N), and saves the validated winner as best.acf. Triggers on "validate result", "extract best config", "Welch's t-test", "is my speedup real", "save best ACF", "pareto front", "claim speedup", "ship config".
when_to_use
- tuner.start() returned and there's a results CSV.
- User wants to ship an ACF to production.
- Reported speedup feels too good to be true.
Don't use when:
- Search hasn't completed (and user hasn't decided to stop early).
- User just wants the raw best row — that's one line of pandas.
The score CompileIQ reports during a search uses N=5-15 trials per evaluation
and a shared cache. That's appropriate for the search loop but wildly
insufficient for shipping. This skill is the gate before any ACF goes to
production.
When
tuner.start() has returned and there's a dump_results= CSV on disk.
User wants to claim a speedup or ship an ACF.
A reported speedup feels too clean — validate it.
Steps
1. Load the CSV
from compileiq.results import SearchResult
results = SearchResult.from_csv("results.csv", problem_type="min", clear_duplicates=True)
df = results.get_results()
print(f"{len(df)} evaluations across {df['generation'].max()+1} generations")
score_1 (with underscore-one) is the canonical key — it matches the
multi-objective convention score_N. Older code sometimes uses plain score;
the fallback above handles both shapes.
For top-K:
import pandas as pd
df_valid = df[pd.to_numeric(df["score_1"], errors="coerce") < 1e10]
top_k = df_valid.nsmallest(5, "score_1") # nlargest for MAX problems
Multi-objective:
front = results.pareto_front() # raises if num_objectives == 1for candidate in front:
print(candidate["score_1"], candidate["score_2"], candidate["params"])
Mixed user+compiler search space: results carry separate keys —
best["user_space"] for the user-side knobs, best["params"] for the ACF
hex. Save both.
3. Re-measure on fresh cache (the actual validation)
Stage
Warmup
Trials
Cache
GPU clocks
Optimization (during tuner.start())
5-25
5-15
per-eval
recommended locked
Validation
≥50
≥100
per-measurement
must be locked
Both the baseline (no ACF) and each top-K candidate are re-measured at
validation N. The optimization-time measurement is too noisy to ship from.
4. Statistical gate — the ship rule
import numpy as np
from scipy import stats
defvalidate_speedup(baseline_ms: np.ndarray, optimized_ms: np.ndarray) -> dict:
t, p = stats.ttest_ind(baseline_ms, optimized_ms, equal_var=False) # Welch's
b_mean, b_std = baseline_ms.mean(), baseline_ms.std(ddof=1)
o_mean, o_std = optimized_ms.mean(), optimized_ms.std(ddof=1)
pooled = np.sqrt((b_std**2 + o_std**2) / 2)
d = (b_mean - o_mean) / pooled if pooled > 0else0.0return {
"speedup_mean": b_mean / o_mean,
"speedup_median": np.median(baseline_ms) / np.median(optimized_ms),
"p_value": float(p),
"cohens_d": float(d),
"significant": bool(p < 0.05and o_mean < b_mean and d > 0.2),
"baseline": {"mean": b_mean, "std": b_std,
"p5": np.percentile(baseline_ms, 5),
"p95": np.percentile(baseline_ms, 95)},
"optimized": {"mean": o_mean, "std": o_std,
"p5": np.percentile(optimized_ms, 5),
"p95": np.percentile(optimized_ms, 95)},
}
Ship rule:p_value < 0.05 AND cohens_d > 0.2 (preferably > 0.5)
AND optimized.mean < baseline.mean. Anything weaker, do not claim a
speedup.
5. Three false-positive patterns to actively check
#
Pattern
Symptom
Cause
Check
Disposition
1
Lucky-min
Optimized min is lower but mean is equal or worse
Optimizer picked a config that occasionally runs fast
Compare means, not minimums; reject if optimized.mean ≥ baseline.mean
Reject.
2
Higher-variance
Optimized p5-p95 range is wider than baseline with same mean
ACF didn't speed anything up; just spread the distribution
Compute (p95 - p5) for both; reject if optimized range is materially wider (>25%)
Reject.
3
Multiple-comparisons
Best of 500 evaluations looks 2-5% faster but doesn't reproduce
With 500 evals some will look good by chance
Re-measure top-K on a fresh cache and fresh trials; reject candidates that don't survive
Reject.
6. Save the validated winner
from compileiq.utils.helpers import save_compiler_config
save_compiler_config("best.acf", best["params"])
# Mixed search spaces: persist the user_space knobs separatelyif"user_space"in best:
import json
Path("best.user_space.json").write_text(json.dumps(best["user_space"], indent=2))
7. Reproducibility log
Append one row per candidate decision to validation-log.csv. Fields, per
docs/flashinfer_booster.md:135-148:
timestamp (UTC ISO 8601)
ACF filename + sha256
manifest / release version
benchmark command
GPU model + driver version
CTK version (nvcc release)
ptxas, nvcc paths + versions
framework version or commit (Triton / Helion / FlashInfer / cuTeDSL)
input shape
baseline mean ± std
candidate mean ± std
p-value
Cohen's d
decision: KEPT or REJECTED:<reason>
The scripts/welch_validate.py helper records the timing/statistical fields,
ACF hash, benchmark commands, GPU/toolchain metadata, and common environment
variables automatically. Pass --manifest, --framework, and --input-shape
for workload-specific fields the helper cannot infer.
Prints KEPT or REJECTED:<reason> and appends a row to the log. Also
importable: from welch_validate import validate_speedup.
Self-test
python scripts/welch_validate.py --self-test
Synthesizes two identical normal distributions, asserts the statistical gate
returns significant=False. Then differs them, asserts significant=True.
Catches misconfigured scipy/numpy before a real validation.
Gotchas
pareto_front() raises if num_objectives == 1. Guard with
if results.num_scores > 1: or use try/except.
score_1 vs score. Current API is score_1. Some older results
exporters used plain score. The defensive read pattern
best.get("score_1", best.get("score")) handles both.
Don't validate on the same cache the search used. With CIQ_KEEP_CACHE=1
active during search, validation must explicitly wipe ~/.cache/compileiq
or use a fresh TRITON_CACHE_DIR and HELION_SKIP_CACHE=1. Otherwise the
optimization-time numbers re-appear and you're not validating anything.
Validation N is independent of optimization N. Even if the search used
N=5 per evaluation, validation needs N ≥ 100. Don't try to be clever and
reuse search-time samples.
Next
If the validated speedup ships: commit best.acf and validation-log.csv.
If validation fails: compileiq-debug for diagnosis.
For more thorough exploration: re-run compileiq-run-search with bigger
pool_size/generations.
