| name | publish-results |
| description | Use when you have a raw suite results JSON to turn into a published dataset shard (convert, dry-run, publish with the write token), or when RANKINGS.md needs updating after a publish. Covers the --kind/--suite selection and the mandatory same-PR RANKINGS rule. |
Publish results
Turns a raw suite JSON into a validated Parquet shard on the
JacobPEvans/mlx-benchmarks dataset. Canonical flow:
RUNBOOK Step 5.
Steps
-
Dry-run first — always. Validates against schema.json and plans the
shard path; no network:
.venv/bin/mlx-bench-publish <results.json> --kind <lm-eval|agentic|promptstack|vllm> --suite <suite> --dry-run
--kind selects the converter; --suite must be in the schema enum. The
envelope always validates inside publish() — never pass --no-validate on
a real run.
-
Publish with the write token. The ambient HF_TOKEN is read-only;
publishing needs the Doppler write token:
doppler run -p "$AI_DOPPLER_PROJECT" -c "$AI_DOPPLER_CONFIG" -- \
.venv/bin/mlx-bench-publish <results.json> --kind <...> --suite <...> --hostname <host>
Optional run-context flags — --env-class, --concurrency, --serving-* —
are documented in docs/schema.md.
-
Ranking duty (same PR). After every publish, update the model's row in
RANKINGS.md, pulling numbers back from the dataset
(loop in Keeping this page current).
The page must never drift from the dataset.
Never
Never discard a completed run — publish it with tags.caveat=... and file an
issue instead of throwing away benchmark time.
