name	tts-research-workflow
description	nanoAudioLLM TTS research workflow for model, objective, sampler, text encoder, mel processing, or data-path changes; validate with static checks, overfit analysis, full-run monitoring, Whisper WER, and final eval before trusting a change.

TTS Research Workflow

Use this skill whenever changing nanoAudioLLM training logic, model architecture, sampler behavior, text conditioning, mel processing, optimizer settings, eval logic, or dataset code.

1. Change Code Surgically

Keep the implementation minimal and direct. Do not add abstractions for one-off branches.
Keep the release path on LJSpeech, flow matching, Euler sampling, BigVGAN-compatible log-mels, frozen text encoder, duration predictor, and Whisper WER.
Update configs/docs only when the behavior actually changes.
Before training, run:

.venv/bin/python -m py_compile train.py model.py config.py audio.py data.py evaluator.py metric.py sample.py scripts/*.py
.venv/bin/python scripts/model_size.py --config configs/ljspeech.yaml
.venv/bin/python scripts/model_size.py --config configs/ljspeech_overfit64.yaml
.venv/bin/python scripts/model_size.py --config configs/ljspeech.yaml --flops --batch_size 2 --text_len 128 --seq_len 128

If mel extraction, audio parameters, tokenizer, or dataset packing changed, recalculate stats:

.venv/bin/python scripts/dataset_stats.py --config configs/ljspeech.yaml

2. Overfit First

Run the 64-sample LJSpeech overfit before any full run:

CUDA_VISIBLE_DEVICES=0 .venv/bin/python train.py \
  --config configs/ljspeech_overfit64.yaml

The overfit config must use eval_prompts: overfit, so WER is measured on the same 64 repeated LJSpeech prompts loaded dynamically from data_dir. Do not use an unrelated prompt set or a tiny canary for the overfit WER gate.

Overfit analysis must include:

final/latest train loss, flow loss, and duration loss
WER on all 64 overfit prompts
generated samples from the overfit checkpoint
whether predicted duration is plausible for those prompts
any obvious mel/vocoder mismatch before starting a full run

Sample fixed prompts from the overfit checkpoint:

.venv/bin/python sample.py --ckpt .artifacts/local/ljspeech_byt5_overfit64/latest.pt --text "Hi, how are you?" --steps 50 --cfg_scale 2.0 --out .artifacts/local/generated_samples/overfit_hi.wav
.venv/bin/python sample.py --ckpt .artifacts/local/ljspeech_byt5_overfit64/latest.pt --text "The quick brown fox jumps over the lazy dog." --steps 50 --cfg_scale 2.0 --out .artifacts/local/generated_samples/overfit_fox.wav
.venv/bin/python sample.py --ckpt .artifacts/local/ljspeech_byt5_overfit64/latest.pt --text "Today is a good day to test speech synthesis." --steps 50 --cfg_scale 2.0 --out .artifacts/local/generated_samples/overfit_tts.wav

3. Full Run

Only start the full LJSpeech run after the overfit loss drops clearly and fixed prompt samples are intelligible enough to justify the run.

Start the standard run:

./run.sh

The current full config uses:

configs/ljspeech.yaml
output directory .artifacts/local/ljspeech_byt5_flow_matching_200k
google/byt5-small
effective batch size 64
200K max steps
in-loop small-set WER eval every 1000 steps
eval_num_samples: 8
eval_steps: 50
eval_cfg_scale: 2.0
Whisper on CUDA

4. Monitor Loss And WER

During a full run, monitor:

train/loss, train/flow_loss, train/duration_loss
eval/wer every eval interval
optim/lr
perf/samples_per_sec and time/step_sec
checkpoint cadence and latest.pt

Useful checks:

tail -n 20 .artifacts/local/ljspeech_byt5_flow_matching_200k/metrics.jsonl
.venv/bin/python scripts/plot_training_loss.py --metrics .artifacts/local/ljspeech_byt5_flow_matching_200k/metrics.jsonl --out .artifacts/local/loss_wer.png
.venv/bin/python scripts/plot_step_time.py --metrics .artifacts/local/ljspeech_byt5_flow_matching_200k/metrics.jsonl

If listening samples are needed mid-run, pause cleanly with the run STOP marker, wait for latest.pt to be written, sample, then resume with the same config. Do not kill the process if a clean pause is possible.

5. Full Eval

Run final small-set eval from the selected checkpoint. Use the same small prompt set used during monitoring unless the experiment explicitly defines another LJSpeech-local eval set.

.venv/bin/python scripts/eval_checkpoint.py \
  --ckpt .artifacts/local/ljspeech_byt5_flow_matching_200k/latest.pt \
  --prompts <small_eval_prompts.txt> \
  --out_dir .artifacts/local/eval_small_final \
  --limit 64 \
  --steps 50 \
  --cfg_scale 2.0 \
  --whisper_model base.en \
  --whisper_device cuda

Generate a few listening samples that are not from the eval list:

.venv/bin/python sample.py --ckpt .artifacts/local/ljspeech_byt5_flow_matching_200k/latest.pt --text "Hi, how are you?" --steps 50 --cfg_scale 2.0 --out .artifacts/local/final_samples/hi.wav
.venv/bin/python sample.py --ckpt .artifacts/local/ljspeech_byt5_flow_matching_200k/latest.pt --text "The weather is beautiful today." --steps 50 --cfg_scale 2.0 --out .artifacts/local/final_samples/weather.wav
.venv/bin/python sample.py --ckpt .artifacts/local/ljspeech_byt5_flow_matching_200k/latest.pt --text "A small model can still learn to speak clearly." --steps 50 --cfg_scale 2.0 --out .artifacts/local/final_samples/small_model.wav

Reporting

Always report:

changed files and validation commands
checkpoint path
latest/full-run loss and WER
path to the loss/WER plot
final eval wer.jsonl and summary.json
sample wav paths
any blocked check with the exact missing command, dependency, or artifact

name	tts-research-workflow
description	nanoAudioLLM TTS research workflow for model, objective, sampler, text encoder, mel processing, or data-path changes; validate with static checks, overfit analysis, full-run monitoring, Whisper WER, and final eval before trusting a change.