voice

name	voice
description	VoiceCLI assistant — author TTS scripts, generate speech, clone voices, transcribe audio, manage samples. Knows engine capabilities, markdown format, and all CLI commands. Triggers: "voice" | "voicecli" | "speech" | "generate speech" | "clone voice" | "transcribe" | "TTS" | "text to speech" | "voice script".
version	0.6.0
allowed-tools	Read, Edit, Write, Bash, Glob, Grep

VoiceCLI — Unified Voice Assistant

Purpose: Author TTS markdown scripts, run voicecli CLI commands, manage the full voice pipeline.

VoiceCLI is a Python CLI for local voice generation (TTS) and transcription (STT) with multiple AI backends.

Auto-Discovery

Before the first command in a session, determine how to invoke voicecli:

# 1. Preferred: voicecli in PATH (installed via `voicecli init`)
if command -v voicecli &>/dev/null; then
  echo "voicecli is in PATH — use it directly"
# 2. Fallback: find project dir and use uv run
else
  if [ -n "$VOICECLI_DIR" ] && [ -f "$VOICECLI_DIR/src/voicecli/cli.py" ]; then
    true  # already set via env var
  else
    for d in . .. ../voiceCLI ~/projects/voiceCLI; do
      test -f "$d/src/voicecli/cli.py" && VOICECLI_DIR="$(cd "$d" && pwd)" && break
    done
  fi
  [ -z "$VOICECLI_DIR" ] && echo "voiceCLI not found — run 'voicecli init' or set VOICECLI_DIR" && exit 1
  echo "Using fallback: cd $VOICECLI_DIR && uv run voicecli"
fi

Cache the result for the session:

• If in PATH: use voicecli <args> directly (works from any directory)
• Fallback: use cd "$VOICECLI_DIR" && uv run voicecli <args>

All examples below use voicecli — substitute with the fallback form if needed.

First Use

On the first invocation of any voice/TTS command in a session, proceed directly with the request.

voicecli doctor is a diagnostic/installation tool — only run it when the user explicitly asks for a system check or when troubleshooting an error. Never run it automatically.

Engine Capability Matrix

Five engines with different strengths. Pick based on user intent:

Capability	Qwen	Qwen-Fast	Chatterbox Multilingual	Chatterbox Turbo	Voxtral
instruct (free-form emotion)	yes	yes	no	no	no
segments (per-section overrides)	yes	yes	yes	yes	yes
paralinguistic tags	to_instruct	to_instruct	strip	native	strip
exaggeration (0.25–2.0)	no	no	yes (per-segment)	yes (per-segment)	no
cfg_weight (0.0–1.0)	no	no	yes (per-segment)	yes (per-segment)	no
flow_steps (3=fast, 8=quality)	no	no	no	no	yes (per-segment)
cfg_alpha (1.0–1.2)	no	no	no	no	yes (per-segment)
temperature (0.0–2.0)	yes	yes	yes	yes	no
top_p (0.0–1.0)	yes	yes	yes	yes	no
min_p (0.0–1.0)	no	no	yes	yes	no
repetition_penalty	yes	yes	yes (def 2.0)	yes (def 1.2)	no
language (multi)	yes (per-segment)	yes (per-segment)	yes (per-segment)	no (EN only)	yes (9 langs)
built-in voices	yes (9, per-segment)	yes (9, per-segment)	no	no	yes (20 presets)
voice cloning	yes	yes	yes	yes	no
CUDA graph acceleration	no	yes (5-9x speedup)	no	no	no
VRAM usage	~3.5 GB	~3.5 GB	~1.8 GB	~1.8 GB	~3.7 GB (int4)

Engine Selection Guide

User wants…	Engine
Multi-language + voice cloning	chatterbox
English + emotion tags ([laugh], [sigh])	chatterbox-turbo
Free-form emotion instructions	qwen or qwen-fast
Named built-in voices (9 Qwen)	qwen or qwen-fast
Many voice presets (20, 9 languages)	voxtral
Per-section emotion changes	qwen or qwen-fast
Per-section expressiveness	chatterbox or chatterbox-turbo
Bilingual content (language switching)	qwen, qwen-fast, chatterbox, or voxtral
Fastest generation (CUDA graph accel.)	qwen-fast
High-quality multilingual (no cloning needed)	voxtral

Tag Handling

Tags: [laugh] [chuckle] [cough] [sigh] [gasp] [groan] [sniff] [shush] [clear throat]

• Turbo: tags kept in text, engine processes them natively
• Qwen: tags split into segments with mapped instruct (e.g. [laugh] → instruct: "Laughing")
• Multilingual: tags stripped (use exaggeration/cfg_weight instead)

The code translator handles this automatically — write tags in the universal format.

Qwen Voices

Vivian, Serena, Uncle_Fu, Dylan, Eric, Ryan (default), Aiden, Ono_Anna, Sohee

Voxtral Voices

20 presets across 9 languages with auto-selection by language:

neutral_female (default), neutral_male, cheerful_female, casual_female, casual_male, fr_male, fr_female, de_male, de_female, es_male, es_female, it_male, it_female, pt_male, pt_female, nl_male, nl_female, ar_male, hi_male, hi_female

When language is set (e.g. language: French), Voxtral auto-resolves to the matching voice (e.g. fr_female). Explicit voice overrides this.

User Config (voicecli.toml)

Optional file for default settings. Discovery: voicecli walks up from the CWD to $HOME looking for voicecli.toml — place it in a parent directory (e.g. ~/projects/voicecli.toml) to share it across multiple projects. A warning is printed to stderr if no file is found anywhere.

[defaults]
language = "French"
engine = "qwen"
accent = "Léger accent du sud provençal"
personality = "Voix calme, douce et flamboyante"
exaggeration = 0.7
cfg_weight = 0.3
segment_gap = 200
crossfade = 50
# plain = false         # strip [tags] and ignore <!-- directives -->
# chunked = false       # always output separate chunk files
# chunk_size = 500      # target chunk size in chars (~15 chars/sec)

Structured instruct parts (accent, personality, speed, emotion) auto-compose into instruct. Raw instruct bypasses composition. Write instruct parts in the target language.

Segment propagation: toml structured parts are backfilled into .md segments where frontmatter didn't set them, so a script with no frontmatter still inherits instruct from voicecli.toml.

# ── Voxtral-specific defaults (also settable via --flow-steps / --cfg-alpha) ──
# flow_steps = 8               # ODE solver steps (3=fast, 8=quality)  [Voxtral]
# cfg_alpha = 1.2              # classifier-free guidance (1.2=quality, 1.0=faster)  [Voxtral]
# temperature = 0.8            # sampling diversity  [Qwen/Chatterbox]
# top_p = 1.0                  # nucleus sampling  [Qwen/Chatterbox]
# min_p = 0.05                 # min probability threshold  [Chatterbox]
# repetition_penalty = 2.0     # repeated token penalty  [Qwen/Chatterbox]

Priority: CLI flag > markdown frontmatter > voicecli.toml > hardcoded default

Unified Markdown Format

One .md file works across all engines — the code translator adapts per engine.

Frontmatter (all optional)

---
language: French          # language name (Qwen + Chatterbox Multilingual)
voice: Ryan               # built-in voice (Qwen only)
engine: qwen              # qwen | qwen-fast | chatterbox | chatterbox-turbo | voxtral
accent: "Provençal"       # pronunciation/origin (Qwen, composes into instruct)
personality: "Calme"      # character traits (Qwen, composes into instruct)
speed: "Rythme posé"      # tempo/pace (Qwen, composes into instruct)
emotion: "Chaleureuse"    # emotional state (Qwen, composes into instruct)
instruct: "Parle avec colère" # raw instruct bypass (overrides structured parts)
exaggeration: 0.75        # expressiveness 0.25-2.0 (Chatterbox only)
cfg_weight: 0.3           # speaker adherence 0.0-1.0 (Chatterbox only)
flow_steps: 8             # ODE solver steps, 3=fast 8=quality (Voxtral only)
cfg_alpha: 1.2            # classifier-free guidance, 1.0=faster 1.2=quality (Voxtral only)
temperature: 0.8          # sampling diversity (Qwen/Chatterbox, default 0.8)
top_p: 1.0                # nucleus sampling (Qwen/Chatterbox, default 1.0)
min_p: 0.05               # min probability threshold (Chatterbox only, default 0.05)
repetition_penalty: 2.0   # repeated token penalty (Qwen/Chatterbox, default varies)
segment_gap: 200          # ms silence between segments (default 0)
crossfade: 50             # ms fade between segments (default 0)
---

Per-Section Directives

All frontmatter fields can be overridden per-section using <!-- key: value --> comments. Multiple keys can be combined in a single comment: <!-- emotion: "Passionnée", speed: "Rapide" -->. Directives accumulate before a text block and apply to the text that follows. Each section inherits frontmatter defaults, overridden by its inline directives. Commas inside quoted values are safe: <!-- emotion: "Passionnée, mais contenue" -->.

Available: accent, personality, speed, emotion, instruct, exaggeration, cfg_weight, flow_steps, cfg_alpha, temperature, top_p, min_p, repetition_penalty, language, voice, segment_gap, crossfade

Segment Transitions

gap	crossfade	Result
0	0	Direct concat (default)
>0	0	Hard cut, silence, hard cut
0	>0	Fade-out then fade-in (no silence)
>0	>0	Fade-out, silence, fade-in

Body Features

• Standard markdown (auto-stripped to plain text)
• <!-- emotion: "Passionnée", speed: "Rapide" --> — multi-key on one line (preferred)
• <!-- accent: "Parisien" --> — single-key still works
• <!-- instruct: "Speak seriously" --> — raw instruct bypass (Qwen)
• <!-- exaggeration: 0.8, cfg_weight: 0.3 --> — per-section expressiveness (Chatterbox)
• <!-- flow_steps: 8, cfg_alpha: 1.2 --> — per-section Voxtral quality tuning
• <!-- temperature: 0.6, repetition_penalty: 1.5 --> — per-section sampling (Qwen/Chatterbox)
• <!-- language: Japanese, voice: Ono_Anna --> — per-section language + voice switch
• <!-- segment_gap: 500, crossfade: 100 --> — per-section transition control
• [laugh] [sigh] etc. — paralinguistic tags (see tag handling above)

Example Script

---
language: French
accent: "Léger accent provençal"
personality: "Calme et douce"
emotion: "Chaleureuse"
exaggeration: 0.7
cfg_weight: 0.3
segment_gap: 200
---

Welcome everyone. [laugh] This is going to be fun!

<!-- emotion: "Passionnée", segment_gap: 500 -->
Now let me tell you something important. [sigh] It has been a long road.

<!-- language: Japanese, voice: Ono_Anna, crossfade: 100, segment_gap: 0 -->
A section in Japanese with a different voice, crossfaded in.

Recommended Settings

• Passionate speech: exaggeration 0.7–0.8, cfg_weight 0.3
• Cross-language cloning: cfg_weight 0.0 (reduces accent bleed)
• Default: exaggeration 0.5, cfg_weight 0.5
• Natural pauses between sections: segment_gap 200–300

CLI Commands Reference

Generate (built-in voices)

voicecli generate "Hello world"                       # Qwen, default voice
voicecli generate "Bonjour" -e chatterbox --lang French
voicecli generate "text" -e chatterbox-turbo          # English + tags
voicecli generate "text" -e qwen-fast                 # CUDA-accelerated Qwen
voicecli generate "Bonjour" -e voxtral                # Voxtral (auto voice by lang)
voicecli generate script.md                           # from markdown
voicecli generate article.txt                         # from plain text file
voicecli generate script.md --mp3                     # + MP3 output
voicecli generate "text" -v Ono_Anna --lang Japanese  # specific voice
voicecli generate script.md --segment-gap 300         # 300ms between segments
voicecli generate script.md --crossfade 50            # 50ms fade transitions
voicecli generate script.md --plain                   # strip [tags] and ignore <!-- directives -->
voicecli generate "text" --fast                       # 0.6B model (faster, lower quality)
voicecli generate "Long text" --chunked               # progressive output (separate files)
voicecli generate "Long text" --chunked --chunk-size 300  # smaller chunks (~20s each)
voicecli generate article.txt --chunked               # long plain text file → chunks
voicecli generate "Bonjour" --via-nats                # route through hub TTS satellite (no local model load)
voicecli generate script.md --via-nats --timeout 90   # ditto, with custom NATS request timeout

--via-nats (or VOICECLI_VIA_NATS=1) sends the request to the hub satellite over NATS and writes the returned WAV locally — useful from hosts without GPU/model. Requires NATS_URL + an NKey seed. Full guide: docs/NATS-SERVE.md#client-side.

Clone (voice cloning)

voicecli clone "text" --ref voice.wav                 # clone from audio
voicecli clone "text"                                 # uses active sample
voicecli clone script.md --mp3                        # from markdown + MP3
voicecli clone article.txt --chunked                  # from plain text file → chunks
voicecli clone "text" -e chatterbox --lang French     # multilingual clone
voicecli clone "text" -e qwen-fast                    # CUDA-accelerated clone
voicecli clone script.md --segment-gap 200            # with segment transitions
voicecli clone script.md --plain                      # strip [tags] and ignore <!-- directives -->
voicecli clone "text" --fast                          # 0.6B model (faster, lower quality)
voicecli clone "Long text" --chunked                  # progressive output (separate files)

Transcribe (speech-to-text)

voicecli transcribe audio.wav                         # default model
voicecli transcribe audio.wav --json                  # JSON + timestamps
voicecli transcribe audio.wav -m large-v3             # specific model
voicecli transcribe audio.wav -l fr                   # force language

Models: tiny, base, small, medium, large-v3, large-v3-turbo (default)

Listen (real-time mic STT)

voicecli listen                                       # Kyutai 1b (EN+FR)
voicecli listen -m 2.6b                               # EN-only, higher quality

Samples

voicecli samples list                                 # list all samples
voicecli samples add file.wav                         # import WAV
voicecli samples record name -d 30                    # record 30s from mic
voicecli samples use name.wav                         # set active for clone
voicecli samples active                               # show current active
voicecli samples remove name.wav                      # delete sample

Daemon (warm model — fast generation)

voicecli serve                                        # start daemon (lazy-loads on first request)
voicecli serve --engine qwen                          # preload Qwen at startup (~60s once, then ~2-5s per call)
voicecli serve --engine qwen --fast                   # preload smaller 0.6B model

generate and clone automatically route to the daemon when it's running — no extra flags. Falls back silently to standalone if the socket isn't present.

NATS Serve (distributed queue-group satellite)

voicecli nats-serve {tts,stt} runs voicecli as a long-lived NATS queue-group subscriber instead of a local Unix-socket daemon — the hub and voicecli talk over NATS subjects rather than a local socket, so they can live on different hosts and be restarted independently.

voicecli nats-serve tts                               # TTS satellite (queue group: tts_workers)
voicecli nats-serve stt                               # STT satellite (queue group: stt_workers)
voicecli nats-serve tts --allow-coexist               # bypass VRAM guard (large-VRAM hosts only)
voicecli nats-serve stt --model large-v3-turbo        # override STT model

When to prefer NATS vs Unix socket:

Scenario	Use
Hub and voicecli on the same host, no network boundary	voicecli serve / stt-serve (Unix socket)
Hub and voicecli on different hosts	voicecli nats-serve tts / nats-serve stt
Add satellite capacity without changing hub config	nats-serve (add queue-group members)
Production with hub managed separately from GPU worker	nats-serve

Environment variables (read at startup — restart to apply):

Variable	Default	Purpose
NATS_URL	— (required)	NATS server URL, e.g. nats://lyra-nats:4222
NATS_NKEY_SEED_PATH	— (required for nkey auth)	Path to NKey seed file — must be 0600
NATS_CA_CERT	— (optional)	PEM CA certificate for TLS verification
VOICECLI_ENGINE / LYRA_TTS_ENGINE	from voicecli.toml	TTS engine override (qwen, qwen-fast, chatterbox, …)
VOICECLI_MODEL	large-v3-turbo	STT model name (STT-only)
VOICECLI_MAX_CONCURRENT	TTS 1 / STT 2	Parallel request slots — keep at 1 on single-GPU hosts
VOICECLI_HEARTBEAT_INTERVAL	5.0	Seconds between heartbeats (must stay ≤ 5)
VOICECLI_DRAIN_TIMEOUT	30	Graceful-shutdown drain window (exit 3 if exceeded)
VOICECLI_REJECT_WHEN_FULL	unset	1 → reject new requests when all slots busy
VOICECLI_ALLOW_COEXIST	unset	1 → bypass VRAM-sequencing guard (dev/large-VRAM hosts)

Full operator reference: docs/NATS-SERVE.md.

Dictate (STT daemon + overlay)

voicecli dictate                                      # start STT daemon + waveform overlay
voicecli dictate cancel                               # cancel current recording
voicecli dictate next-mode                            # cycle STT mode
voicecli dictate status                               # show daemon status

WSL2 shortcuts (via AHK script): Alt+Shift+Space = toggle, Alt+Shift+Tab = next-mode, Alt+Shift+Esc = cancel. Auto-paste: enabled by auto_paste = true in [stt] section of voicecli.toml.

Dictate NATS (cross-host toggle)

Routes mic audio to a remote STT satellite. Toggle: first call starts, second call stops + transcribes + clipboard.

# Requires env vars:
# NATS_URL=nats://192.168.1.16:4222
# NATS_NKEY_SEED_PATH=~/.roxabi/voicecli/nkeys/voice-client.seed
voicecli dictate nats                                 # toggle recording
voicecli dictate nats --lang fr                       # force French
voicecli dictate nats --model large-v3                # override STT model
voicecli dictate nats --paste                         # auto-paste after transcription

Bind ~/.local/bin/voicecli-dictate-nats (wrapper with env vars) to a global shortcut. On COSMIC (Pop!_OS): System Settings → Keyboard → Custom Shortcuts. Full setup: docs/STT-SETUP.md#nats-dictation.

Utilities

voicecli mp3 TTS/voices_out/file.wav                  # WAV → MP3 (192kbps)
voicecli mp3 TTS/voices_out/file.wav -b 320           # 320kbps
voicecli voices                                       # list Qwen voices
voicecli voices -e chatterbox                         # list engine voices
voicecli engines                                      # list engines
voicecli emotions                                     # emotion cheat sheet
voicecli doctor                                       # system readiness check
voicecli init                                         # create starter voicecli.toml

Project Layout

TTS/
  texts_in/       — authored .md scripts (git-tracked)
  voices_out/     — generated WAV/MP3 (gitignored)
  samples/        — voice samples for cloning (gitignored)
STT/
  audio_in/       — audio files to transcribe (gitignored)
  texts_out/      — transcription results (gitignored)

Chunked Output

Use --chunked for long texts to enable progressive sending via Telegram. Each chunk is saved as a separate numbered file (prefix_001.wav, prefix_002.wav...) and a .done sentinel is written when complete.

• For plain text: splits at paragraph/sentence boundaries (~500 chars per chunk by default)
• For .md scripts with segments: each segment becomes a chunk
• --chunk-size N adjusts target chunk size in characters (~15 chars/sec of speech)

Always use --chunked when generating from Telegram so the bot can send audio progressively.

Output format is WAV by default. Do NOT add --mp3 unless the user explicitly asks for it.

Workflow

Intent → Action Mapping

1. User wants to create a speech → Author a .md script in TTS/texts_in/, then run generate (default) or clone (only if explicitly requested)
2. User wants quick speech → Run generate with inline text
3. User explicitly wants voice cloning → Ensure sample exists (samples list/samples add), then clone. Never default to clone — the presence of an active sample does NOT imply the user wants cloning.
4. User wants to transcribe → Run transcribe on audio file
5. User wants to edit a script → Read existing .md, apply changes respecting the format
6. User asks about capabilities → Consult engine matrix, recommend engine

Script Authoring Rules

Before writing any script, read voicecli.toml to discover the user's configured defaults (engine, language, accent, personality, etc.). Only add frontmatter fields that override those defaults — never duplicate what toml already sets.

When writing .md scripts:

1. Use all features freely — the translator adapts per engine (unsupported fields are nulled per-segment)
2. Use <!-- key: value, key2: value2 --> directives for per-section overrides — combine multiple keys on one line (accent, personality, speed, emotion, instruct, exaggeration, language, voice, segment_gap, crossfade)
3. Place [tags] naturally in the text flow — before or after the relevant phrase
4. Keep text segments under ~250 chars for Chatterbox engines (they chunk at sentence boundaries, but shorter is safer)
5. For multilingual content, set language: in frontmatter and override per-section with <!-- language: ... -->
6. Scripts go in TTS/texts_in/ by convention
7. Use segment_gap and crossfade in frontmatter for global defaults, override per-section for fine control
8. Write all instruct parts (accent, personality, speed, emotion, instruct) in the target language

Telegram Integration

When running inside the 2ndBrain Telegram bot, voice files auto-send to the user's chat after generation. Always use --chunked so the bot can send audio progressively. Skip send_voice_telegram.py — just generate with --chunked and report what you created.

--plain Flag

Use --plain on generate or clone to strip all [tags] and ignore <!-- directives -->:

• On a .md file: frontmatter settings (language, voice, instruct) are still applied; only per-section directives and tags are ignored
• On raw text or .txt file: tags are stripped from the text
• Can be set permanently in voicecli.toml as plain = true

Key Constraints

• Qwen clone does NOT support instruct — only generate does
• Chatterbox Turbo is English-only
• Chatterbox Multilingual strips all paralinguistic tags
• Voxtral does NOT support voice cloning (codec encoder not released by Mistral)
• Voxtral does NOT support instruct, exaggeration, or cfg_weight — voice character comes from preset selection
• Voxtral (3.7 GB VRAM) cannot coexist with the Qwen daemon (8.5 GB) on 16 GB GPU — stop daemon first
• Both Chatterbox engines have a ~40s generation cutoff (handled by auto-chunking per segment)
• Clone falls back to active sample when --ref is omitted
• qwen-fast has same capabilities as qwen but uses CUDA graph acceleration (5-9x speedup after warmup)
• --fast flag uses the smaller 0.6B model (Qwen/Qwen-fast only) — faster but lower quality
• Base instruct is preserved in tag-split segments (e.g. [laugh] on Qwen keeps the original instruct alongside the tag instruct)
• Daemon (voicecli serve) only accelerates Qwen engines — Chatterbox and Voxtral always run standalone
• Daemon socket: ~/.local/share/voicecli/daemon.sock; if absent, generate/clone fall back silently

$ARGUMENTS