omni-bench-publish

name

omni-bench-publish

description

Publish a scored omni-bench result (ASR or text-generation) to a leaderboard platform (e.g. bench.bshk.app) via its write API. Use when asked to publish, upload, submit, or "send a report" for an omni-bench result.json or parity-report.json — sharing ASR WER/RTFx numbers or text-generation TTFT/tok-per-s/prompt-cache numbers. Covers the one modality-agnostic `omni-bench publish` command and the MANDATORY secret handling: the api-key is injected from a secrets manager (1Password `op run` / AgentVault `av run`) at runtime, never read into context, never hardcoded.

Publishing omni-bench results (ASR or text generation)

Upload a scored result.json (and optionally a parity-report.json) to an omni-bench results platform (default https://bench.bshk.app) via POST /api/v1/results. The command is identical for every modality — ASR (WER / RTFx) and text generation (tok/s / TTFT / prompt-cache): the scorer already wrote the correct row shape and the platform detects the modality from it.

Prerequisites

• A scored document from omni-bench score (publish the result.json, never a raw run-artifact).
• An api-key for the platform, held in a secrets manager (see the secret-handling section — do NOT paste it anywhere).
• The platform base URL.

The command (same for ASR and text-gen)

omni-bench publish reads the key from $OMNI_BENCH_API_KEY; ownership is taken from the key's principal server-side, never from the document body.

omni-bench publish --result runs/<run>.result.json [--parity runs/<run>.parity.json] --url https://bench.bshk.app
# → published result: <resultId> ({"resultId": …, "identityKey": "sha256:…"})

Add --private to stage the upload owner-only (x-omni-visibility: private).

Secret handling — inject the key, NEVER read or hardcode it

Never echo/cat the key, never pass --api-key <literal> in scripts, never write it to a committed file. Inject it into the publish command's environment at runtime so the value never enters your shell or context. In order of preference:

• 1Password (op run) — recommended. Put an op:// reference (not the value) in an env-file:

  printf 'OMNI_BENCH_API_KEY=op://<vault>/<item>/api-key\n' > .env.omni
  op run --env-file .env.omni -- omni-bench publish --result runs/x.result.json --url https://bench.bshk.app
  

  If the field lives in a section, the reference includes it (op://<vault>/<item>/<section>/api-key); run op item get <item> --format json and copy the field's reference verbatim.
• AgentVault (av run). av add OMNI_BENCH_API_KEY (value entered at the TTY prompt, never an argument); an agentvault.yaml maps a profile to { ref: av://file/OMNI_BENCH_API_KEY, tier: normal }; then av run --profile <p> -- omni-bench publish --result runs/x.result.json --url https://bench.bshk.app.
• CI / other manager. Expose OMNI_BENCH_API_KEY in the publish step's env only — e.g. GitHub Actions env: { OMNI_BENCH_API_KEY: ${{ secrets.OMNI_BENCH_API_KEY }} }.

op read / av read print a raw value — they are human-at-a-TTY tools (they refuse pipes/files), not for the agent. Always use the run wrappers, which resolve the reference into the subprocess env and keep the value out of your context.

Full flow (either modality)

omni-bench prepare <task_id>          # e.g. asr.fleurs.ru.v1  OR  textgen.shared_prefix.en.v1
# Run YOUR host adapter over the manifest → run-artifact.jsonl:
#   Python: omni-bench run --adapter pkg.mod:factory --manifest data/<task>/manifest.json --out runs/x.jsonl
#   Swift:  Producer.run(transcriber: …)   (ASR)   |   Producer.run(generator: …)   (text-gen)
omni-bench score --artifact runs/x.jsonl --references data/<task>/references.jsonl --out runs/x.result.json
op run --env-file .env.omni -- omni-bench publish --result runs/x.result.json --url https://bench.bshk.app

Never reimplement scoring in a host — the centralized Python scorer is the trust anchor; emit hypotheses/generations and let it grade them.

Response + verify

• 201 { resultId, identityKey } on success.
• 401 wrong/missing key · 409 run_id already owned by another principal (or an identity race — retry) · 413 document > 4 MB · 422 schema (unsupported schema_version, or a degenerate row with no sortable metric).
• Verify on the board: <url>/leaderboard?board=asr (ASR) or ?board=textgen (text generation); the run's detail at <url>/result/<identityKey>.

Notes

• Schema is pinned; current omni-bench emits 0.1.0 (re-cut 2026-07-04 — the earlier pre-release 0.1.0/0.2.0 shapes are retired and rejected with 422; the platform accepts only the re-cut 0.1.0).
• Identity fields (model.sha256, backend.build_sha, hardware.soc/mem_gb, run_profile.decode) feed identity_key — fill them honestly or the parity join / dedupe breaks.
• A text-gen result with no references legitimately has quality: null — it still publishes (the ingest guard rejects only a row with no headline metric: ASR needs quality, text-gen needs tok_per_s or ttft_s_p50).