| name | release |
| description | mlx-serve pre-release validation checklist, CalVer versioning, release steps, and CHANGELOG style. Use when preparing or cutting a release, running pre-release validation, or writing CHANGELOG entries. |
Pre-release validation — ALWAYS run this, same process every time
Timings measured 2026-07-16 on the M4 Max 128 GB, AFTER the stop_all_engines port-wait fix (before it, everything below was ~2.2× slower — see the gotcha in Benchmarking).
| # | Step | Command | Time |
|---|
| 1 | Hermetic suite | zig build test (must be 6/6 steps, 0 fail) + cd app && swift test | ~1 min |
| 2 | ReleaseFast binary | zig build -Doptimize=ReleaseFast → du -h zig-out/bin/mlx-serve ≈ 7 MB (Debug ≈ 2× = fake regression) | ~10 s |
| 3 | Perf gate (did WE regress?) | ./tests/bench.sh (mlx-serve only, llmprobe) → diff vs the previous column in benchmarks.md → append this release's column | ~15 min |
| 4 | Tool-call correctness | zig build test -Dtest-filter="format corpus" + -Dtest-filter="tool traffic"; live: ./tests/test_tool_matrix_small.sh | ~3 min |
| 5 | API conformance | npx llmprobe@latest http://127.0.0.1:<port>/v1 --quick → expect 100% engine conformance | ~10 s/model |
| 6 | Regression scripts | integration_test.sh, test_anthropic_api.sh, test_ollama_api.sh, test_stream_keepalive.sh, test_disconnect_cancel.sh, test_pld_equivalence.sh, test_mtp_equivalence.sh | ~15 min |
| 7 | Soak (bigger releases) | SOAK_DURATION_HOURS=1 ./tests/test_soak_24h.sh — RSS drift < 10% | 1 h |
| 8 | Cross-engine table (only when an ENGINE version changed, or before a public claim) | start each engine yourself, ./tests/bench.sh --url <host:port> -m <id> --full per engine → rewrite the vs other engines table in benchmarks.md | ~90 min |
| 9 | Bundle | SKIP_NOTARIZE=1 bash app/build.sh (both binaries move together) | ~2 min |
Rules:
- Steps 3 and 8 are different questions. 3 = "did our code regress" — mlx-serve only, the ONLY one needed every release. 8 = the public comparison; LM Studio/oMLX/MTPLX numbers cannot move when only OUR code changes, so re-run 8 only when an engine version bumps.
- Diff step 3 against llmprobe columns only. Columns through 26.7.12 are the pre-2026-08 hand-rolled bench, a DIFFERENT methodology — frozen history, never a diff target. See /bench.
--only <substr> runs a single model row for tight dev loops.- Depth: default
--bench-only is one run per ladder rung to 16k. --full takes median-of-3 per rung and climbs to 32k/64k — that's the release artifact depth (step 8). For a regression CLAIM on a spec-decode cell, sample across runs and boot orders regardless of depth: "reproducible ≠ not variance".
- Never quote a win without naming the engine it is over — vs LM-GGUF the 26B-A4B row reads +33%; vs oMLX it is +1.6%.
benchmarks.md gets one new COLUMN per release, from the rows step 3 prints (Laguna's number comes from its own A/B harness, not the bench matrix). Obey the file's own header rules: results into the tables only, no text; Apple M4 Max 128 GB only — skip the update entirely when releasing from any other machine (the M4 mini), a mixed column poisons the history.
Release benchmark artifacts
The record is benchmarks.md plus the saved llmprobe reports under ~/claude-tmp/bench-<tag>/. Nothing lands in docs/ any more — no CSVs, no charts. Run the gate on the FINAL release tree (a number taken mid-cycle is stale the moment another perf round lands):
./tests/bench.sh --tag <ver>
Rules:
- Paste the printed rows into the
Decode tok/s by release table, one new column, mode suffix included. A cell that lost its mode suffix is the signal that speculation stopped engaging — chase it before shipping.
--full takes median-of-3 per rung and climbs to 32k/64k; the default is one run per rung to 16k. For a regression CLAIM on a spec cell, sample across runs and boot orders regardless of depth.- The one chart left is
docs/perf-vs-engines.png, frozen at the release it was rendered for and named as such in the README caption. There is no longer a script that regenerates it; refresh the vs other engines table instead.
Versioning & Releases
CalVer YY.M.N (e.g., v26.4.25 = 2026, April, 25th release). N auto-increments from the last GitHub release for that YY.M prefix; build.sh computes via gh release list.
Version sources: app/Info.plist (CFBundleVersion/CFBundleShortVersionString), Zig -Dversion build option (build_options.version), git tag (gh release create v{version}). CI derives ONE version and stamps all three — the bundle plist is stamped from it, never shipped as committed (v26.8.1's DMG reported 26.7.12 and nagged forever; docs/gotchas/app.md).
YY.M is TZ-pinned (America/New_York, in release.yml + app/build.sh): runners are UTC, so a workflow_dispatch cut after ~20:00 local otherwise rolls into next month. Prefer a tag push over a dispatch when the version is already decided — the tag-push path takes version=${GITHUB_REF_NAME#v} and never consults the clock.
Release:
- Update
CHANGELOG.md with NEXT version (check gh release list --limit 1 first — never reuse an existing tag)
- Dont commit or push
CHANGELOG style
One entry per shipped release. No new entries for unshipped work — fold it into the next pending entry. Always run gh release list --limit 1 first; if the topmost CHANGELOG entry is newer than the latest GitHub release, that entry is unshipped and any new bullets get merged into it. Never bump version numbers ahead of an actual release.
Tone: high-level executive bullets, marketing-style. The audience is users/integrators, not contributors reading the diff.
- Lead each bullet with what changed for the user (capability, speed, model support), not the implementation.
- Quantify where impressive — concrete tok/s percentages, model names, the workload it applies to.
- Avoid: file paths, function names, internal symbol renames, line-count diffs, "we discovered that…", PR/issue numbers.
- 4–7 bullets per release. If you need more, the release is too big and should ship sooner.
Template:
## vYY.M.N — Two-to-five-word headline
- **<User-visible thing>**: one or two sentences on the impact. Numbers if you have them.
- **<New model / API / behavior>**: what unlocks, when it kicks in, what stays the same.
- **<Speed or reliability win>**: workload + measured gain.
- **<Removed / deprecated thing, if any>**: why, and what users should do instead.
---
When in doubt, look at the existing entries (v26.5.4 and earlier) — keep the same density and tone.
