| name | devloop-bench-cycle |
| description | Full benchmark cycle for devloop — collect criterion results, check against budgets, commit the run to SQLite, surface regressions. Use after running cargo bench or when managing performance budgets. |
Devloop Bench Cycle
Full workflow for collecting, checking, and committing benchmark results in devloop.
Prerequisites
devloop binary installed at ~/.local/bin/devloop- devloop project is the current working directory (or use
--repo)
Step 1 — Run benchmarks
cd ~/dev/devloop
cargo bench --workspace 2>&1 | tee /tmp/bench-output.txt
Criterion outputs JSON results to target/criterion/. The bench cycle reads from there.
Step 2 — Collect results
devloop bench collect 2>&1
This reads criterion output, parses mean_ns, median_ns, std_dev_ns, ci_lower_ns, ci_upper_ns per benchmark, and saves a run to the SQLite bench store.
If collect fails with "no criterion data": ensure cargo bench was run first and target/criterion/ exists.
Step 3 — Check against budgets
devloop bench check 2>&1
Output shows regression status per benchmark:
✓ STABLE — within CI bounds of previous run⚠ WARNING — above previous CI upper bound (statistical regression)✗ REGRESSION — over budget (max_ns exceeded)↓ IMPROVED — below previous CI lower bound
If regressions are found: do not commit the run. Investigate root cause first.
Step 4 — Review history
devloop bench history --last 10 2>&1
Shows trend per benchmark. Useful for distinguishing noise from real regressions.
Step 5 — Manage budgets (if needed)
Set a new budget:
devloop bench budget set <benchmark-name> --max 500ms --warn 400ms
List all budgets:
devloop bench budget list
Remove a budget:
devloop bench budget remove <benchmark-name>
Budget values use time units: 100ns, 1ms, 500ms, 2s.
Step 6 — Commit results
After a clean check (no regressions):
git add -A
git commit -m "bench: collect run on <branch>"
If there were regressions that were investigated and accepted:
git commit -m "bench: collect run — <benchmark-name> regression accepted, reason: <explanation>"
Regression response guide
| Status | Action |
|---|
| IMPROVED | Note in commit message, check it's not a measurement artifact |
| WARNING (+5–20%) | Investigate — check for recent changes to the hot path |
| WARNING (>20%) | Treat as regression — find root cause before proceeding |
| REGRESSION (over budget) | Block — must fix or explicitly raise the budget with justification |
Category guidance
Benchmarks are auto-categorized:
micro — < 1ms mean (e.g., pure computation, data structure ops)integration — >= 1ms mean (e.g., full pipeline runs, I/O-bound ops)
Integration benchmarks get budget enforcement; micro benchmarks use statistical comparison only.
