synthesize-vocal-with-diffsinger

name	synthesize-vocal-with-diffsinger
description	Use when vocal_events.json is ready and a sung vocal WAV needs to be generated via a DiffSinger backend. Handles phonemization and produces rough_vocal.wav. Prefers the vendored Nishiren DiffSinger v2.0 ONNX voicebank.
allowed-tools	Read Bash Grep Glob Write
argument-hint	["vocal-events-json"]
effort	high

Skill 3: synthesize_vocal_with_diffsinger

Purpose

Convert vocal_events.json into the input format expected by the chosen singing backend, then run inference to produce rough_vocal.wav.

This skill combines backend-specific input generation and rendering.

Inputs

vocal_events.json

Config:

{
  "backend": "diffsinger",
  "model_path": "models/diffsinger/singer_a",
  "language": "English",
  "output": "rough_vocal.wav",
  "debug_output": "diffsinger_input.json"
}

Tools

Phonemizer / G2P

Use language-specific tools to convert lyrics/syllables to phonemes.

For English, possible tools:

• g2p-en
• phonemizer
• pronouncing
• fallback LLM correction for sung pronunciation

Nishiren DiffSinger v2.0 (preferred in this repo)

This repo vendors a self-contained ONNX voicebank under:

third_party/Nishiren Diffsinger v2.0/

Use it via:

./bin/synthesize_vocal_with_diffsinger vocal_events.json \
  --backend nishiren_onnx \
  --nishiren-root "third_party/Nishiren Diffsinger v2.0" \
  --nishiren-lang en \
  --nishiren-style Standard \
  --out rough_vocal.wav \
  --debug-out diffsinger_input.json \
  --log synthesis_log.json

This backend requires real phonemes; in the current implementation it uses a minimal built-in lexicon for the Old MacDonald demo and should be extended (or replaced by a real G2P) for general lyrics.

DiffSinger (OpenVPI-style)

Use a DiffSinger inference script/checkpoint. DiffSinger is score-conditioned; practical inference inputs usually include text/phoneme sequence, note sequence, note durations, and slur flags.

The exact schema is fork/checkpoint-specific. This skill should hide that backend messiness.

Internal conversion

From backend-independent events:

{
  "pitch": "C4",
  "duration_beats": 1.0,
  "lyric": "Twin",
  "is_slur": false
}

To DiffSinger-style input:

{
  "text": "twinkle twinkle little star",
  "ph_seq": "t w ih n k ax l t w ih n k ax l l ih t ax l s t aa r",
  "note_seq": "C4 C4 D4 D4 E4 E4 G4",
  "note_dur_seq": "0.50 0.50 0.50 0.50 0.50 0.50 1.00",
  "is_slur_seq": "0 1 0 1 0 1 0",
  "input_type": "phoneme"
}

Duration conversion:

duration_seconds = duration_beats * 60 / tempo_bpm

Outputs

rough_vocal.wav
diffsinger_input.json
synthesis_log.json

Why this output is useful

rough_vocal.wav is the first actual sung audio.

diffsinger_input.json is required for debugging:

• pronunciation problems → inspect phonemes
• rhythm problems → inspect durations
• melody problems → inspect notes
• bad held syllables → inspect slur flags

Suggested CLI

./bin/synthesize_vocal_with_diffsinger vocal_events.json \
  --model models/diffsinger/singer_a \
  --language English \
  --out rough_vocal.wav \
  --debug-out diffsinger_input.json \
  --log synthesis_log.json

Implementation notes

The first implementation can support one DiffSinger fork/checkpoint. Later, add adapter classes:

DiffSingerAdapterBase
├── MoonInTheRiverAdapter
├── OpenVPIAdapter
└── CustomCheckpointAdapter

Each adapter should implement:

build_input(vocal_events) -> backend_input
run_inference(backend_input, model_path, output_wav) -> synthesis_log

Failure modes

Fail if:

• model path is missing
• phonemizer cannot process the language
• note and duration sequence lengths do not match
• DiffSinger inference exits nonzero
• output WAV is not produced

Warn if:

• unknown words required fallback phonemization
• phoneme count and note count required aggressive slur insertion
• output is clipped or silent