ffmpeg-usage

name	ffmpeg-usage
description	Make polished videos with ffmpeg: a production pipeline plus vetted recipes for conversion, scaling, compositing, GIFs, captions, audio/voiceover, motion, and platform-ready export — for demos, ads, tutorials, launches, social clips, explainers, and pitches.

ffmpeg Usage

Overview

Two things in one skill: a command library of vetted ffmpeg recipes for everyday media jobs, and a production pipeline for making videos that actually look intentional. Treat ffmpeg as the final compositor and encoder — the renderer — and bring in helper tools for the parts ffmpeg is clumsy at (asset design, animation, captions, 3D). Use it whenever a request touches media: converting, resizing, building GIFs, pulling audio, editing, or producing a finished video for marketing, demos, tutorials, launches, social, explainers, app-store previews, or pitches.

Version: 2.0.0 Needs: ffmpeg ≥ 4.0; ffprobe recommended. Other tools (below) are optional and used only when they earn their place.

When to apply

Trigger on requests to:

• Convert, rescale, or re-fit aspect ratio
• Build a GIF, extract or convert audio
• Edit clips — trim, merge, speed, rotate
• Work with subtitles or captions — burn-in, soft, extract, generate
• Compress, web-optimise, or format for a platform (YouTube, Instagram, TikTok, X)
• Grab thumbnails or frames; batch-process a folder
• Plan and produce a finished video — story beats, scene composition, animated text, framing, voiceover/music, and a repeatable render pipeline

Prerequisites

brew install ffmpeg              # macOS
sudo apt-get install ffmpeg      # Debian/Ubuntu
choco install ffmpeg             # Windows
ffmpeg -version                  # confirm

---

Command library

1. Format conversion

MP4 → WebM:

ffmpeg -i input.mp4 -c:v libvpx-vp9 -crf 30 -b:v 0 -c:a libopus output.webm

MOV → MP4:

ffmpeg -i input.mov -c:v libx264 -c:a aac -strict experimental output.mp4

Anything → MP4 (broad compatibility):

ffmpeg -i input.* -c:v libx264 -preset medium -crf 23 -c:a aac -b:a 128k output.mp4

2. Resolution

Rescale while preserving aspect ratio.

To 720p / 1080p:

ffmpeg -i input.mp4 -vf scale=-1:720 -c:a copy output_720p.mp4
ffmpeg -i input.mp4 -vf scale=-1:1080 -c:a copy output_1080p.mp4

Fixed width, auto height:

ffmpeg -i input.mp4 -vf scale=1280:-1 -c:a copy output.mp4

Fit with letterbox padding:

ffmpeg -i input.mp4 -vf "scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2" output.mp4

3. GIFs

Palette generation is what keeps GIFs sharp and small.

Quick (10 fps):

ffmpeg -i input.mp4 -vf "fps=10,scale=480:-1:flags=lanczos" output.gif

Two-pass with palette:

ffmpeg -i input.mp4 -vf "fps=10,scale=480:-1:flags=lanczos,palettegen" palette.png
ffmpeg -i input.mp4 -i palette.png -filter_complex "fps=10,scale=480:-1:flags=lanczos[x];[x][1:v]paletteuse" output.gif

From a time range (single pass):

ffmpeg -ss 00:00:10 -t 5 -i input.mp4 -vf "fps=10,scale=480:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" output.gif

4. Audio

Extract to MP3 / WAV:

ffmpeg -i input.mp4 -vn -acodec libmp3lame -q:a 2 output.mp3
ffmpeg -i input.mp4 -vn -acodec pcm_s16le -ar 44100 -ac 2 output.wav

Convert format:

ffmpeg -i input.wav -c:a aac -b:a 192k output.m4a

Replace audio with a track / mix two together:

ffmpeg -i video.mp4 -i music.mp3 -c:v copy -c:a aac -map 0:v:0 -map 1:a:0 -shortest output.mp4
ffmpeg -i video.mp4 -i music.mp3 -filter_complex "[0:a][1:a]amix=inputs=2:duration=first" -c:v copy output.mp4

5. Editing

Trim (stream-copy, no re-encode):

ffmpeg -i input.mp4 -ss 00:00:10 -to 00:00:30 -c copy output.mp4   # 10s–30s
ffmpeg -i input.mp4 -ss 00:00:05 -t 10 -c copy output.mp4          # 10s from 5s

Concatenate — pick by container:

# Protocol — TS/MPEG/MP3/AAC, no list file needed:
ffmpeg -i "concat:file1.mp3|file2.mp3|file3.mp3" -c copy output.mp3

# Demuxer — MP4/MOV/MKV, via process substitution:
ffmpeg -f concat -safe 0 -i <(printf "file '%s'\n" video1.mp4 video2.mp4 video3.mp4) -c copy output.mp4
# Fallback if the shell lacks process substitution:
printf "file '%s'\n" video1.mp4 video2.mp4 video3.mp4 > list.txt
ffmpeg -f concat -safe 0 -i list.txt -c copy output.mp4 && rm list.txt

# Filter — mismatched codecs/resolutions, re-encodes:
ffmpeg -i video1.mp4 -i video2.mp4 -i video3.mp4 \
  -filter_complex "[0:v][0:a][1:v][1:a][2:v][2:a]concat=n=3:v=1:a=1[v][a]" \
  -map "[v]" -map "[a]" output.mp4

Decision: MP3/AAC/TS/MPEG → protocol; MP4/MOV/MKV → demuxer; differing codecs or sizes → filter.

Speed / rotate:

ffmpeg -i input.mp4 -filter:v "setpts=0.5*PTS" -an output.mp4   # 2x
ffmpeg -i input.mp4 -filter:v "setpts=2.0*PTS" output.mp4       # 0.5x
ffmpeg -i input.mp4 -vf "transpose=1" output.mp4               # 90° CW
ffmpeg -i input.mp4 -vf "transpose=2,transpose=2" output.mp4   # 180°

6. Subtitles

ffmpeg -i input.mp4 -vf subtitles=subtitles.srt output.mp4       # burn in
ffmpeg -i input.mp4 -i subtitles.srt -c copy -c:s mov_text output.mp4   # soft subs
ffmpeg -i input.mp4 -map 0:s:0 subtitles.srt                     # extract

7. Thumbnails

ffmpeg -i input.mp4 -ss 00:00:05 -vframes 1 thumbnail.jpg   # one frame
ffmpeg -i input.mp4 -vf fps=1/10 thumb%04d.jpg              # every 10s
ffmpeg -i input.mp4 -vframes 10 frame%04d.png              # first 10 frames

8. Compression & web

ffmpeg -i input.mp4 -c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k output.mp4    # balanced
ffmpeg -i input.mp4 -c:v libx264 -crf 28 -preset veryslow -c:a aac -b:a 96k output.mp4   # smaller
ffmpeg -i input.mp4 -c:v libx264 -preset medium -crf 23 -movflags +faststart -c:a aac -b:a 128k output.mp4   # web

Platform presets

YouTube:

ffmpeg -i input.mp4 -c:v libx264 -preset slow -crf 18 -c:a aac -b:a 192k -pix_fmt yuv420p -movflags +faststart youtube.mp4

Instagram Story (9:16, ≤15s):

ffmpeg -i input.mp4 -vf "scale=1080:1920:force_original_aspect_ratio=decrease,pad=1080:1920:(ow-iw)/2:(oh-ih)/2" -c:v libx264 -preset medium -crf 23 -c:a aac -b:a 128k -t 15 instagram_story.mp4

X / Twitter (16:9, ≤2:20):

ffmpeg -i input.mp4 -vf scale=1280:720 -c:v libx264 -preset medium -crf 23 -c:a aac -b:a 128k -t 140 twitter.mp4

TikTok (9:16, ≤60s):

ffmpeg -i input.mp4 -vf "scale=1080:1920:force_original_aspect_ratio=decrease,pad=1080:1920:(ow-iw)/2:(oh-ih)/2" -c:v libx264 -preset medium -crf 23 -c:a aac -b:a 128k -t 60 tiktok.mp4

Quick recipes

# Shrink a screen recording
ffmpeg -i screen_recording.mov -c:v libx264 -preset medium -crf 23 -vf "scale=1920:-1" -c:a aac -b:a 128k optimised.mp4

# Batch-convert a folder
for i in *.mov; do ffmpeg -i "$i" -c:v libx264 -crf 23 -c:a aac "${i%.mov}.mp4"; done

# Images → video
ffmpeg -framerate 30 -pattern_type glob -i '*.jpg' -c:v libx264 -pix_fmt yuv420p output.mp4   # sequence
ffmpeg -loop 1 -i image.jpg -c:v libx264 -t 5 -pix_fmt yuv420p output.mp4                      # 5s from one image

---

Polished video production workflow

For good-looking videos, use ffmpeg as the final compositor/encoder, not as the only creative tool.

1. Define the story beats. One clear idea per beat: hook, problem, solution, proof, call to action. Write the through-line before touching a command.
2. Gather source media — screen recordings, product shots, images, AI/stock clips, or generated backgrounds. Capture clean footage: use a release build, hide debug overlays, cursors, and notifications, and record at the target resolution or higher.
3. Normalise all media to the target format — usually 1080x1920 vertical or 1920x1080 landscape, a consistent frame rate (30 fps is a safe default), and yuv420p.
4. Create reusable assets — transparent text overlays, frames, masks, backgrounds, captions, logos.
5. Compose scene-by-scene with ffmpeg, one short clip per beat.
6. Add motion — pan, zoom, scroll, parallax, easing, transitions.
7. Add audio — voiceover, music, sound effects, captions, and loudness normalisation.
8. Render preview frames / contact sheets before the final export.
9. Verify with ffprobe.

Prefer several short scenes over one long continuous recording — easier to time, replace, review, and reuse.

Normalise clip timing with setpts=(target/actual)*PTS, e.g. retime an 8s clip to 6s:

ffmpeg -y -i clip-raw.mp4 -vf "setpts=0.75*PTS,fps=30,format=yuv420p" -an clip-normalised.mp4

Useful libraries and tools

ffmpeg is excellent for encoding, compositing, filters, audio mixing, scaling, and final export. Other tools are better at generating assets, animation, captions, and visual design. Reach for them only when ffmpeg gets awkward.

Visual composition

• ffmpeg — final composition, encoding, A/V sync, transitions, overlays, scaling, cropping, filters.
• ImageMagick — quick image conversion, resizing, trimming, masks, contact sheets.
• Pillow / PIL — generate transparent PNG text overlays, caption cards, thumbnails, title frames, gradients, simple layouts.
• OpenCV — frame analysis, crop detection, motion tracking, masking, scene processing.
• PySceneDetect — auto-detect scene changes in long videos.
• Blender — 3D product/device renders, camera movement, lighting, shadows.
• Three.js / React Three Fiber — web-based 3D scenes and animated product mockups.
• Remotion — React-based programmatic video; reusable branded templates and dynamic videos.
• Manim — animated diagrams and technical/educational explainers.
• Lottie / Bodymovin — lightweight vector animations from After Effects or JSON.

Selection rule:

• ffmpeg for the final render and media processing.
• Pillow / HTML+CSS for designed text overlays.
• Remotion when the layout is mostly UI, components, and text.
• Blender or Three.js when the video needs real 3D depth.
• OpenCV when it needs detection, tracking, cropping, or frame analysis.

Text, captions, and typography

For premium text, prefer generating transparent PNG overlays over relying only on ffmpeg drawtext (which is fragile around quoting and font discovery).

• Pillow — reliable text rendering, wrapping, shadows, gradients, tracking, caption cards.
• HTML/CSS screenshots via Playwright — best when text needs web-quality typography: flexbox/grid, custom fonts, gradients, responsive layout.
• ASS subtitles — styled and karaoke-style captions.
• SRT/VTT — simple subtitles with broad platform support.
• Whisper / faster-whisper — transcribe voiceover into captions.
• aeneas / Gentle / WhisperX — forced alignment for accurate word timing.

Guidelines: keep headlines short; keep captions clear of important content; respect mobile safe margins; avoid flashing text unless it's a deliberate, readable rhythm; animate text once, then hold steady; generate preview frames to check overlap and readability. Prefer a condensed bold display font for headlines.

Audio, voiceover, and music

Good audio reads as more professional than extra visual effects.

Voiceover:

• Kokoro TTS / Piper — lightweight local open-source TTS; good for repeatable, private generation.
• Coqui XTTS — expressive TTS and voice cloning; check licensing and model availability before commercial use.
• OpenAI TTS / ElevenLabs / PlayHT — higher-quality hosted voices when commercial polish matters; need API keys and cost control.

Music and sound: use open-licensed music when publishing commercially and keep the licence/attribution file beside the asset; prefer instrumental tracks under voiceover; duck music under speech; add subtle effects for transitions, taps, reveals, and success states; normalise final loudness.

# Normalise loudness
-af "loudnorm=I=-16:TP=-1.5:LRA=11"

# Fade music in and out
-af "afade=t=in:ss=0:d=1,afade=t=out:st=27:d=1"

# Mix voice and music
-filter_complex "[voice][music]amix=inputs=2:duration=first"

# Duck (sidechain) music under voice
-filter_complex "[music][voice]sidechaincompress=threshold=0.04:ratio=8:attack=20:release=250[ducked];[voice][ducked]amix=inputs=2"

# Loop music to fit the video without retiming the picture
ffmpeg -y -i silent.mp4 -stream_loop -1 -i music.mp3 -map 0:v:0 -map 1:a:0 \
  -shortest -c:v copy -c:a aac -b:a 192k -movflags +faststart final.mp4

Voiceover guidance: one short line per scene, matched to what's on screen; use punctuation to control rhythm; nudge TTS speed up slightly if it sounds flat; leave pauses between beats; generate captions from the final voiceover, not the draft script.

Motion and animation

Static compositions feel like slideshows. Add subtle motion to every scene: slow push-in, gentle pan, device parallax, background drift, screen scroll, text ease-in then hold, fast-but-readable transitions on the beat, light zoom for emphasis. Prefer smooth easing over linear movement; animate once at a beat's start, then hold.

# Ease-out slide-in for a text overlay PNG (smooth, no flicker)
ffmpeg -y -i base.mp4 -i title.png \
  -filter_complex "[0:v][1:v]overlay=x='(W-w)/2':y='if(lt(t,0.35),-h+(220+h)*sin(t/0.35*PI/2),220)':enable='between(t,0,3)'[v]" \
  -map "[v]" -map 0:a? -c:v libx264 -crf 18 -preset medium -pix_fmt yuv420p \
  -c:a aac -b:a 192k -movflags +faststart out.mp4

# Ken Burns push-in
zoompan=z='min(zoom+0.0015,1.08)':d=150:x='iw/2-(iw/zoom/2)':y='ih/2-(ih/zoom/2)'

Framing products, devices, and mockups

When footage shows an app, website, or product, frame it so it reads as a real object rather than a flat rectangle.

• Compose on the target canvas (e.g. 1080x1920) and keep text in top/bottom bands, off the footage.
• Fit footage into the screen area, preserving aspect ratio:

  ffmpeg -y -i clip.mp4 -vf "scale=564:1284:force_original_aspect_ratio=decrease,pad=564:1284:(ow-iw)/2:(oh-ih)/2:black,fps=30,format=rgba" screen.mp4
  

• Make the device smaller than the canvas, with rounded body, bezel, notch/island, side buttons, and shadow; keep the content in a transparent screen cutout.
• Give the device a stage — warm rim light, soft shadow, gentle glow, faint outline. Never sit a black device on a black background.
• For consumer work, a cinematic dark background with warm white/gold type usually beats neon accent bars. Pick one accent colour and keep it.
• Accent key words within a line (white text, final keyword in the accent) instead of one flat colour.
• Write beat-specific support copy that explains the moment — never one generic line under every scene.
• Keep texture subtle; a little grain adds depth, visible noise cheapens it.

A solid vertical layout to start from:

canvas: 1080x1920
media stage: x=40, y=235, w=1000, h=1420
device body: w=600, h=1298
device stage: dark rounded rectangle behind the device, low-opacity warm outline
headline: y=72, condensed uppercase, wide tracking
bottom tagline: y=1708, condensed uppercase, final keyword in accent
support copy: y=1814, smaller light text
accent: one warm colour, used consistently

Video QA checklist

Before final delivery:

• ffprobe to confirm dimensions, duration, codecs, pixel format, and audio:

  ffprobe -v error -select_streams v:0 \
    -show_entries stream=width,height,r_frame_rate,duration \
    -show_entries format=duration,size -of default=nw=1 final.mp4
  

• Export preview frames at key timestamps and build a contact sheet for review:

  ffmpeg -y -ss 00:00:03 -i final.mp4 -frames:v 1 -update 1 preview-03.png
  

• Check text readability on mobile and that captions don't overlap important content.
• Check audio starts and ends cleanly and that voiceover matches what's on screen.
• Check for debug overlays, reload flashes, cursor artifacts, or stale screens.
• Confirm the export uses yuv420p and +faststart.

Recommended final export:

ffmpeg -i input.mp4 \
  -c:v libx264 -crf 18 -preset medium -pix_fmt yuv420p \
  -c:a aac -b:a 192k \
  -movflags +faststart \
  output.mp4

---

Best practices

• Inspect first: ffprobe -v quiet -print_format json -show_format -show_streams input.mp4
• -c copy whenever you can to skip re-encoding.
• Test on a slice with -t 10 before committing to a long render.
• CRF guide: 18 ≈ visually lossless, 23 = good default, 28 = smaller and acceptable (range 0–51).
• -movflags +faststart on web video for progressive playback.

Guidelines for the agent

On a media request: identify the task, pick the matching recipe, confirm prerequisites (ffmpeg present, input exists), explain the command, run it with error handling, verify the output, and suggest optimisations where they help. For a finished video, work the pipeline above — beats first, then source, compose, motion, audio, QA — and break complex jobs into explained steps. Reach for a helper tool only when ffmpeg alone is awkward, and say why.

For concatenation, prefer process substitution with printf over a temporary list.txt; fall back to the file only when the shell can't do it.

Always: confirm the input exists and is readable, check ffmpeg is installed, ensure the output path is writable, fail with a clear message, and show progress on long renders.

References

• Docs: https://ffmpeg.org/documentation.html
• Wiki: https://trac.ffmpeg.org/wiki
• Codecs: https://ffmpeg.org/ffmpeg-codecs.html
• Filters: https://ffmpeg.org/ffmpeg-filters.html