omni-bench-run

name	omni-bench-run
description	Run a model through the omni-bench benchmark to MEASURE it — produce a run-artifact and score it, then read the numbers. Use when asked to run, benchmark, measure, evaluate, or "test the results of" a model with omni-bench — ASR (WER/CER, RTFx) or text generation (tok/s, TTFT, prefill tok/s, prompt-cache speedup). Covers the adapter seam (Transcriber/Generator), the prepare→run→score→diff CLI flow, the offline no-download smoke task, and how to interpret each metric. NOT for publishing to a leaderboard (use omni-bench-publish) or changing the framework itself (use omni-bench).
version	1.0.0

Running omni-bench to measure a model

omni-bench measures quality + speed tied to specific hardware (ASR: WER/CER + RTFx; text generation: tok/s, TTFT/prefill, prompt-cache benefit). It is a framework, not a runner: YOU provide inference through a small adapter seam; omni-bench owns the datasets, the scoring, and the metrics. Scoring lives in one place (the Python scorer) so numbers are comparable across languages/backends — never reimplement scoring in your host.

Full producer guide (Python + Swift, copy-paste): omni-bench/docs/integration.md.

The seam — plug in your model

Implement one structural contract (no inheritance; Swift mirrors Python 1:1):

• ASR: Transcriber.transcribe(audio, language, task) -> Transcript (text + optional backend_native.decode_s, peak_mem_gb).
• Text generation: Generator.generate(prompt, task) -> Generation + reset_cache(). The seam is non-streaming, so all latency/token metrics are host-reported via backend_native: ttft_s, prefill_s, prompt_tokens, generated_tokens, reused_tokens, cache_hit, decode_s, peak_mem_gb. The scorer derives the rates — emit raw counts/times, don't compute tok/s yourself.

The flow

omni-bench prepare <task_id>                 # materialize dataset → data/<task_id>/{manifest.json, references.jsonl}
omni-bench run --adapter pkg.mod:factory \   # YOUR host over the manifest → run-artifact.jsonl
    --manifest data/<task_id>/manifest.json --out runs/x.jsonl
omni-bench score --artifact runs/x.jsonl \   # centralized scorer → result.json (the numbers)
    --references data/<task_id>/references.jsonl --out runs/x.result.json
omni-bench diff  --result-a … --artifact-a … --result-b … --artifact-b … --out p.json  # optional: parity of two runs

--adapter is a dotted path module:factory; factory() returns (Transcriber|Generator, model, backend) — model/backend are host-declared identity (base_model_id, artifact_id, sha256, quantization; backend id + build_sha). omni-bench and your model package must import in the same interpreter (uv pip install -e omni-bench/python into your env, or set PYTHONPATH). Swift hosts call Producer.run(transcriber:/generator: …).

Offline smoke (no downloads)

textgen.shared_prefix.en.v1 is fully synthetic (4 groups × 3 prompts, shared ~1k-word prefix) — the only downloads-free task; ASR tasks hit HuggingFace. For a pipeline-only smoke (no real model) point --adapter at a fake Generator returning fixed text + plausible timings; to benchmark a real model, wrap its inference in the seam.

Read the numbers (result.json → rows[0])

• ASR row: quality{wer_norm, wer_ortho, cer} (lower better), speed{rtfx_native, rtfx_wall, peak_mem_gb}. rtfx_native (from decode_s) is cross-implementation comparable; rtfx_wall is harness-scoped (rendered distinct). quality: null ⇒ no sample transcribed (a failed run, not a perfect one).
• Text-generation row: main_score: "tok_per_s"; speed{tok_per_s, prefill_tok_per_s, ttft_s_p50, prefill_s_p50, peak_mem_gb}; cache{cold, warm, ttft_speedup, reused_tokens_mean, cache_hit_rate}|null; quality{exact_match}|null (null when the task ships no references — the synthetic scenario does).
  • prefill_tok_per_s = cache-free prompt-processing rate (cold samples only in the shared-prefix scenario; the llama-bench pp equivalent).
  • ttft_speedup = cold.ttft_s_p50 / warm.ttft_s_p50 (> 1 ⇒ the prompt cache helped) — the canonical single-number cache benefit. Nothing is discarded as warmup; cold is the baseline.
• Sanity: counts{n_ok, n_error, n_timed} — a run with n_error > 0 had per-sample failures (the producer records them and never aborts).

Rules that keep results honest

• Never reimplement scoring in the host — emit hypotheses/generations + raw counts, let the one Python scorer grade them (this is the whole point: no measurer drift).
• Declare hardware honestly — hardware{soc, mem_gb} keys the identity and speed is hardware-scoped; off macOS the collector can't see the accelerator, so pass it explicitly.
• Deterministic — greedy/temperature 0/seed pinned; prepare twice ⇒ identical hashes.

Next / related

• Share the numbers on a leaderboard → the omni-bench-publish skill (omni-bench publish, api-key injected from a secrets manager).
• Change the framework (schemas, scorer, seams, datasets) → the omni-bench skill.
• just/uv may live outside the default PATH (e.g. /opt/homebrew/bin).