Use this skill when adding audio to programmatic videos - generating narration with ElevenLabs TTS, sourcing royalty-free background music, creating SFX with FFmpeg, implementing audio ducking, or mixing multiple audio layers in Remotion. Triggers on ElevenLabs, text-to-speech, voice generation, background music, sound effects, audio mixing, and volume ducking.
Instrucciones de origen · Vista previa de solo lectura
name
video-audio-design
version
0.1.0
description
Use this skill when adding audio to programmatic videos - generating narration with ElevenLabs TTS, sourcing royalty-free background music, creating SFX with FFmpeg, implementing audio ducking, or mixing multiple audio layers in Remotion. Triggers on ElevenLabs, text-to-speech, voice generation, background music, sound effects, audio mixing, and volume ducking.
Layered audio architecture - Every video has three audio layers:
narration on top (loudest), SFX in the middle (accent volume), and
background music at the base (lowest).
Narration drives timing - Generate narration first, measure its
duration, then set scene timing to match. Never fit narration into
arbitrary scene lengths.
Duck music during speech - Background music must drop 50-60% when
narration plays. Use smooth ramps (10-15 frames) to avoid jarring jumps.
SFX as accents, not distractions - Keep SFX short (under 0.5s),
subtle in volume, and relevant to on-screen action.
Test audio in context - Always preview the full mix with all layers
together. Listen for muddy speech, volume spikes, or dead silence.
Core concepts
3-layer audio architecture
Layer
Role
Base Volume
During Narration
Narration
Conveys information, drives pacing
0.8-1.0
N/A (top layer)
SFX
Accents transitions and actions
0.3-0.5
0.3-0.5 (unchanged)
Background Music
Sets emotional tone, fills silence
0.3-0.5
0.15-0.25 (ducked)
ElevenLabs API model
ElevenLabs provides neural TTS via a REST API. The core flow:
Pick a voice (pre-made or cloned) - each has a voice_id
Send text + voice settings to /v1/text-to-speech/{voice_id}
Receive raw audio bytes (mp3 by default)
Write to file and measure duration for scene timing
Voice settings:
Setting
Range
Low
High
Recommended
stability
0-1
More expressive, variable
More consistent, monotone
0.4-0.6
similarity_boost
0-1
More creative
Closer to original voice
0.6-0.8
style
0-1
Neutral delivery
Exaggerated style
0.3-0.6
Audio ducking concept
Audio ducking reduces background music volume when narration starts and
restores it when narration ends. In Remotion, use interpolate():
Music volume: 0.4 ---\ /--- 0.4
\ /
0.15 \__________/
narration start → end
Ramps should take 10-15 frames (~0.3-0.5s at 30fps).
Frame-based audio sync in Remotion
useCurrentFrame() returns the current frame number
interpolate() maps frame ranges to value ranges (e.g., volume)
<Sequence from={frame}> places audio at a specific frame
<Audio volume={fn}> accepts a static number or a per-frame function
Convert seconds to frames: frames = seconds * fps.
Common tasks
1. Set up ElevenLabs API key and generate narration
Implement audio ducking - drop music 50-60% during speech
Hardcoding ElevenLabs API key
Key leaks into version control
Use environment variables: process.env.ELEVEN_LABS_API_KEY
Using TTS without measuring duration
Scene timing wrong, narration cut off
Measure audio duration with ffprobe after generation
SFX louder than narration
Distracts from content
SFX at 0.3-0.5, narration at 0.8-1.0
No fade on music start/end
Abrupt start/stop sounds like a bug
Add 0.5-1s fade-in at start and fade-out at end
Using low-quality TTS model
Robotic voice undermines quality
Use eleven_multilingual_v2 or tts-1-hd
Ignoring audio file format
Some formats add silence padding
Use MP3 for narration, WAV for SFX
Gotchas
ElevenLabs rate limits and character quotas - The free tier has a monthly character limit. Cache generated audio aggressively and only regenerate when text changes. Use a hash of the text as the cache key.
MP3 encoder padding adds silence - MP3 files often have 20-50ms of silence at the start. Trim with ffmpeg -af silenceremove=1:0:-50dB or account for the offset in frame timing.
Remotion Audio volume is per-component, not global - Two <Audio> components at volume 1.0 can clip. Keep total volume across simultaneous layers under 1.0.
FFmpeg SFX sound different across systems - Always specify -ar 44100 -sample_fmt s16 for consistent output across machines.
Voice consistency across scenes - ElevenLabs can produce different tones for the same settings with varying text. Use stability >= 0.5 for multi-scene narration.
References
For detailed patterns on specific audio sub-domains, read the relevant file
from the references/ folder:
references/elevenlabs-api.md - advanced ElevenLabs API patterns including
voice cloning, streaming TTS, websocket API, pronunciation dictionaries,
and quota management
references/audio-mixing-patterns.md - advanced mixing patterns including
multi-segment ducking, crossfades between scenes, volume automation curves,
and mastering the final mix
references/sfx-generation.md - comprehensive SFX generation with FFmpeg
including complex synthesis, layering multiple generators, and building a
reusable SFX library
Only load a references file if the current task requires it - they are
long and will consume context.
Companion check
On first activation of this skill in a conversation: check which companion skills are installed by running ls ~/.claude/skills/ ~/.agent/skills/ ~/.agents/skills/ .claude/skills/ .agent/skills/ .agents/skills/ 2>/dev/null. Compare the results against the recommended_skills field in this file's frontmatter. For any that are missing, mention them once and offer to install:
