| name | omni-bench-run |
| description | Run a model through the omni-bench benchmark to MEASURE it — produce a run-artifact and score it, then read the numbers. Use when asked to run, benchmark, measure, evaluate, or "test the results of" a model with omni-bench — ASR (WER/CER, RTFx) or text generation (tok/s, TTFT, prefill tok/s, prompt-cache speedup). Covers the adapter seam (Transcriber/Generator), the prepare→run→score→diff CLI flow, the offline no-download smoke task, and how to interpret each metric. NOT for publishing to a leaderboard (use omni-bench-publish) or changing the framework itself (use omni-bench). |
| version | 1.0.0 |
Running omni-bench to measure a model
omni-bench measures quality + speed tied to specific hardware (ASR: WER/CER + RTFx;
text generation: tok/s, TTFT/prefill, prompt-cache benefit). It is a framework, not a
runner: YOU provide inference through a small adapter seam; omni-bench owns the datasets,
the scoring, and the metrics. Scoring lives in one place (the Python scorer) so numbers
are comparable across languages/backends — never reimplement scoring in your host.
Full producer guide (Python + Swift, copy-paste): omni-bench/docs/integration.md.
The seam — plug in your model
Implement one structural contract (no inheritance; Swift mirrors Python 1:1):
- ASR:
Transcriber.transcribe(audio, language, task) -> Transcript (text + optional
backend_native.decode_s, peak_mem_gb).
- Text generation:
Generator.generate(prompt, task) -> Generation + reset_cache().
The seam is non-streaming, so all latency/token metrics are host-reported via
backend_native: ttft_s, prefill_s, prompt_tokens, generated_tokens,
reused_tokens, cache_hit, decode_s, peak_mem_gb. The scorer derives the rates —
emit raw counts/times, don't compute tok/s yourself.
The flow
omni-bench prepare <task_id>
omni-bench run --adapter pkg.mod:factory \
--manifest data/<task_id>/manifest.json --out runs/x.jsonl
omni-bench score --artifact runs/x.jsonl \
--references data/<task_id>/references.jsonl --out runs/x.result.json
omni-bench diff --result-a … --artifact-a … --result-b … --artifact-b … --out p.json
--adapter is a dotted path module:factory; factory() returns
(Transcriber|Generator, model, backend) — model/backend are host-declared identity
(base_model_id, artifact_id, sha256, quantization; backend id + build_sha). omni-bench and
your model package must import in the same interpreter (uv pip install -e omni-bench/python
into your env, or set PYTHONPATH). Swift hosts call Producer.run(transcriber:/generator: …).
Offline smoke (no downloads)
textgen.shared_prefix.en.v1 is fully synthetic (4 groups × 3 prompts, shared ~1k-word
prefix) — the only downloads-free task; ASR tasks hit HuggingFace. For a pipeline-only smoke
(no real model) point --adapter at a fake Generator returning fixed text + plausible timings;
to benchmark a real model, wrap its inference in the seam.
Read the numbers (result.json → rows[0])
- ASR row:
quality{wer_norm, wer_ortho, cer} (lower better), speed{rtfx_native, rtfx_wall, peak_mem_gb}. rtfx_native (from decode_s) is cross-implementation
comparable; rtfx_wall is harness-scoped (rendered distinct). quality: null ⇒ no sample
transcribed (a failed run, not a perfect one).
- Text-generation row:
main_score: "tok_per_s"; speed{tok_per_s, prefill_tok_per_s, ttft_s_p50, prefill_s_p50, peak_mem_gb}; cache{cold, warm, ttft_speedup, reused_tokens_mean, cache_hit_rate}|null; quality{exact_match}|null (null when the task
ships no references — the synthetic scenario does).
prefill_tok_per_s = cache-free prompt-processing rate (cold samples only in the
shared-prefix scenario; the llama-bench pp equivalent).ttft_speedup = cold.ttft_s_p50 / warm.ttft_s_p50 (> 1 ⇒ the prompt cache helped) —
the canonical single-number cache benefit. Nothing is discarded as warmup; cold is the baseline.
- Sanity:
counts{n_ok, n_error, n_timed} — a run with n_error > 0 had per-sample failures
(the producer records them and never aborts).
Rules that keep results honest
- Never reimplement scoring in the host — emit hypotheses/generations + raw counts, let the
one Python scorer grade them (this is the whole point: no measurer drift).
- Declare hardware honestly —
hardware{soc, mem_gb} keys the identity and speed is
hardware-scoped; off macOS the collector can't see the accelerator, so pass it explicitly.
- Deterministic — greedy/temperature 0/seed pinned;
prepare twice ⇒ identical hashes.
Next / related
- Share the numbers on a leaderboard → the
omni-bench-publish skill (omni-bench publish,
api-key injected from a secrets manager).
- Change the framework (schemas, scorer, seams, datasets) → the
omni-bench skill.
just/uv may live outside the default PATH (e.g. /opt/homebrew/bin).
