| name | voxylio-dubbing |
| description | Working on Voxylio's real-time speech/dubbing engine — utterance scheduling, the speech queue, ducking, pacing, cloud (Aura-2) vs local voices, or the stall-recovery nets. Read this before touching the speak/tick/duck paths so the timing invariants and the no-overlap guarantees survive the change. |
Voxylio dubbing engine
Architecture and decisions: docs/DUBBING.md (French). Translation side:
voxylio-translation skill + docs/TRANSLATION.md.
Code map
| Concern | Where |
|---|
| Scheduling: trigger, queue, gates | apps/chrome/src/content.js — tick, onGroupEnter, enqueue, drainQueue |
| Speech engines | speak / speakCloud (Aura-2 MP3 + Audio element) / speakLocal (speechSynthesis) |
| Ducking (speech-gated, dB ramps) | duckNow, maybeReleaseDuck, releaseDuckNow, rampVolumeTo, restoreVolume, onVolumeChange, ctl.onAudioSettings |
| Stall nets, 14s pump, watchdog | the engine-specific block at the top of tick |
| Rate math (adaptive, smoothed) | packages/core/src/pacing.js (computeUtteranceRate, estimateWords) |
| Voice pick + calibration | pickVoice, localWps (content.js); packages/core/src/voices.js |
| Cloud audio cache/prefetch | getCloudAudio, cloudAudioCache, cloudVoiceDownUntil, pretranslate's pre-gen block |
| Rate/buffer events | onRateChange, onBuffering, onPlayingAgain |
Invariants
- One speech slot.
ctl.currentUtterance is the lock for BOTH
engines. The recovery nets are engine-specific: cloud checks
cloudAudio.ended/fetch age; local requires speaking && pending
false AND ≥1.5 s of age (remote voices show idle during startup —
reclaiming early overlapped two voices). Never merge these nets.
- A line starts at its start time, never before.
drainQueue gates
on q.start > t + 0.25 (except during our own autoPause catch-up).
Removing the gate makes the dub run ahead of the picture.
- Ducking is speech-gated with dB-domain ramps. Attack ~250 ms,
release ~700 ms, hold 4.5 s across a burst, no release when a line is
imminent.
writeVolume records lastWrittenVolume; a volumechange we
did not write means the USER took the mix — hands off until they touch
the duck slider (ctl.onAudioSettings) or stop/start. Web Audio API
is NOT an option (cross-origin media tainting) — video.volume only.
- One voice per passage. Cloud refusal/playback failure sets
cloudVoiceDownUntil (+60 s): the passage stays local. Never fall
back per-line.
- Caption + journal at voice time.
speak() shows the caption and
journals via extras ({orig, start, end, rec}) exactly once (rec
survives requeues). Dropped lines are never displayed or counted.
Stats measure onstart→onend; cancelled utterances (_vxCancelled)
count nothing.
- Chrome speechSynthesis quirks are load-bearing: the ~14 s
Google-voice cutoff needs the pause()+resume() pump (cycled 9 s, only
when both functions exist — test stubs lack them); utterance rate is
write-once (ratechange re-schedules local lines, adapts cloud
playbackRate in place via _vxBaseRate); getVoices() is empty at
startup (first line waits ≤1.5 s via ctl.voicesGraceUntil instead of
opening with the wrong voice).
- Pacing: local rate =
computeUtteranceRate with calibrated
localWps (EMA from real utterances, reset on voice change) and
prevRate smoothing; never below the user's base. Cloud rate fits
the MP3's real duration to the slot, clamped ≤1.35, only half the
user's rate preference. QUEUED lines that start >0.8 s late compress
against remaining time; direct lines keep their full window.
- autoPause debts are honoured: resumes the video
it auto-paused (guarded by active/!ended). No path may strand the
user's video paused.
Harness pins (tests/integration) — do not break
run-progressive.js pins the DUCK TIMELINE: some sample < 0.5 during
the dialog AND the last sample ≥ 0.85 after it, plus instant restore
on teardown. It also pins exactly-once speech and no partial versions.run-roll.js pins |u.rate − 1.5| < 0.35 at base 1.0 ×1.5 playback:
direct lines must NOT compress against remaining time (that's what
broke it once), and the base×1.25 catch-up cap must stay.run.js pins strict order, zero duplicates, and that
speechSynthesis.cancel() fires only when we were speaking.- All stubs: no
pause/resume on speechSynthesis (guard with typeof),
no onstart events (stats fall back to call-time), non-empty
getVoices() (the voices-grace path is real-world only).
Gates
pnpm build:chrome
node --test packages/core/test/*.test.js
node tests/integration/run-all.js
pnpm lint:firefox
