bench-regression

name	bench-regression
description	Investigate a Helion benchmark dashboard regression (helionlang.com/dashboard) — find the cause and classify it. Auto-activate when the user reports a perf drop/spike on a dashboard platform (e.g. b200 cute) around a given date.

Goal: explain a dashboard move and sort it into one of four causes — real regression, benchmark method change, hardware issue, or noise. Don't blame a commit until the data points at one.

1. Pull the data, isolate the platform. Numbers are per-nightly, not live — fetch the JSON, don't WebFetch the page (too big to page through):

curl -s https://helionlang.com/dashboard/dashboard-data.json -o /tmp/dash.json

Each summary[] entry has platform_short (e.g. b200_cute) and a history[] of nightlies with helion_speedup_geomean, triton_speedup_geomean, torch_compile_speedup_geomean, helion_latency_avg_ms, sha. Filter to the platform+kernel and print the series around the date.

For the Pretuned tab, use the sibling URL — same structure, different summary[] fields (geomean, best_speedup, helion_wins/total, baselines, cudagraph):

curl -s https://helionlang.com/dashboard/pretuned-dashboard-data.json -o /tmp/pretuned.json

2. Classify — read the whole series, not one point, and sort the move into one of four causes:

• Real regression → Helion genuinely got slower. A code change made the kernel (or the config the autotuner picks) worse. Find the commit (step 3).