| name | readme-benchmarks |
| description | Running the Kokoro-FastAPI benchmark + transcription-roundtrip suites and regenerating the README plots. Use when asked to run/refresh benchmarks, RTF/first-token plots, transcription sanity checks, or the long-form baseline. |
Benchmarks + README plots
Three suites under examples/assorted_checks/ feed the README:
test_transcription/ - synth with a running server, transcribe with faster-whisper, report WER/CER. Short, multilingual, and long-form.benchmarks/ - RTF (processing time vs tokens) and first-token latency/timeline plots. These feed the README performance images.test_dialogue/ - multi-speaker functional checks plus the turn-length / text-length throughput sweeps. Run commands and flags live in test_dialogue/README.md; bench then plot with plot_dialogue_bench.py.
Prereqs
- A Kokoro server on
:8880. GPU and CPU docker images both bind 8880, so swap images, never run both. Set KOKORO_DEVICE / BENCH_PREFIX to match whichever is up.
- Warm the server first. cuDNN autotune cold-start inflates the first GPU run (heavy voices ~1.2s vs ~0.3s warm). Hit a couple voices (e.g.
af_heart, zf_xiaobei) before capturing, or discard the first pass.
examples/ has its own uv venv. Run everything from inside examples/ (cd examples), not the root .venv. The first-token script also needs examples/ as cwd for its audio path.
cd examples
uv sync --extra transcription --extra transcription-gpu --extra benchmarks
Drop --extra transcription-gpu to skip the ~1.2 GB cuDNN/cuBLAS download and transcribe on CPU.
Run (from examples/)
KOKORO_DEVICE=gpu uv run python assorted_checks/test_transcription/test_transcription.py
KOKORO_DEVICE=gpu uv run python assorted_checks/test_transcription/test_transcription_multilingual.py
LONGFORM_CHARS=65000 WHISPER_DEVICE=cuda KOKORO_DEVICE=gpu \
uv run python assorted_checks/test_transcription/test_long_form.py
BENCH_PREFIX=gpu uv run python assorted_checks/benchmarks/benchmark_tts_rtf.py
BENCH_PREFIX=gpu uv run python assorted_checks/benchmarks/benchmark_first_token_stream_unified.py
For a CPU capture: swap to the CPU image, then rerun with KOKORO_DEVICE=cpu / BENCH_PREFIX=cpu (and WHISPER_DEVICE=cpu for long-form). Long-form Windows wrapper: run_long_form.bat [short|full|synth|transcribe].
Env vars
| Var | Applies to | Notes |
|---|
KOKORO_DEVICE | transcription | Label only (server doesn't self-report device). Sets report filename report_{device}.json + meta header so a cpu capture can't overwrite a gpu baseline. Default gpu. |
BENCH_PREFIX | benchmarks | Prefixes RTF output files and picks the token sweep (gpu: dense to 1000; cpu: dense to 500). RTF default gpu, first-token default cpu. |
WHISPER_DEVICE / WHISPER_COMPUTE | transcription | Transcribe device is independent of synth device. Scripts default cpu/int8; long-form .bat uses cuda/float16. |
WHISPER_MODEL | transcription | base.en short, small multilingual. |
LONGFORM_CHARS / LONGFORM_VOICE / LONGFORM_INPUT | long-form | Char cap, voice, input book. |
Gotchas
- First-token plots save under GENERIC names, not
BENCH_PREFIX. BENCH_PREFIX only changes plot titles and the (misleading) "Results saved to" print. Actual files are first_token_{timeline,latency}_stream{,_openai}.png. So a GPU run overwrites the CPU generic file. To keep both, copy the generic file to its device-named asset between runs (CPU run to copy to GPU run).
- Reports are split
_gpu / _cpu with a meta header. Don't let a cpu run clobber a gpu file. If you see a bare report.json / long_form_report.json, it predates the split.
transcribe_seconds is constant across kokoro cpu/gpu unless you change WHISPER_DEVICE. Only synth_seconds tracks the kokoro device.- Regression bands live in
test_transcription/BASELINE.md (WER < 0.07, synth >= 25x rt warm, transcribe >= 40x CUDA / 13-17x CPU int8).
- Replotting doesn't need a server.
{prefix}_benchmark_results_rtf.json keeps processing_time + output_length at full precision, so you can rebuild any RTF plot from the saved data (recompute rtf = processing_time / output_length, feed plot_correlation). No re-synth. Note the stored rtf field is rounded to 2 decimals for the stats text, too coarse to plot, recompute it.
- RTF gets a mean line, not a regression.
plot_correlation(..., show_trend=False) for the RTF plot: rtf is ~constant vs input size, so a fitted slope over-reads single-run noise (each token count is n=1). Processing-time keeps show_trend=True, it's genuinely linear (corr ~0.99). First-token plots use plot_timeline and impose no trend at all.
Output paths
| Suite | Files |
|---|
| short | test_transcription/output/report_{device}.json + wavs |
| multilingual | test_transcription/output_multilingual/report_{device}.json |
| long-form | test_transcription/output_long_form/long_form_report_{device}.json + wav + transcript + *.synth_meta.json sidecar |
| RTF | benchmarks/output_plots/{prefix}_{processing_time,realtime_factor,system_usage}_rtf.png; data in benchmarks/output_data/{prefix}_benchmark_{results,stats}_rtf.* |
| first-token | benchmarks/output_plots/first_token_{timeline,latency}_stream{,_openai}.png, total_time_latency_stream{,_openai}.png |
README asset mapping
Copy plot outputs to assets/ under the README's names (No stamp):
| README asset | Source plot |
|---|
assets/gpu_processing_time.png | output_plots/gpu_processing_time_rtf.png |
assets/gpu_realtime_factor.png | output_plots/gpu_realtime_factor_rtf.png |
assets/gpu_first_token_timeline_openai.png | output_plots/first_token_timeline_stream_openai.png (GPU run; name drops "stream") |
assets/cpu_first_token_timeline_stream_openai.png | output_plots/first_token_timeline_stream_openai.png (CPU run) |
assets/gpu_dialogue_turn_length.png | test_dialogue/output/dialogue_turn_length.png (GPU run; KOKORO_DEVICE only labels the caption) |
assets/gpu_dialogue_text_length.png | test_dialogue/output/dialogue_text_length.png (same run as above) |
The README shows all six across three blocks (grep the asset name to find each). Other gpu_first_token_* assets exist but aren't displayed.
