video-captioning

name	video-captioning
description	Complete video captioning workflow using Whisper. Removes silence/dead zones for fast-paced social media style, generates captions, polishes for errors, creates custom thumbnails with text overlays, embeds subtitles, prepends thumbnail as first frame for platform previews, and generates social media content. Use when user needs to caption and prepare videos for X/LinkedIn/social publishing.

Video Captioning Workflow

This skill automates the complete video captioning process using OpenAI Whisper, from silence removal to transcription to thumbnail generation to platform-ready final video.

Workflow Steps

1. Remove Silence (Auto-Cut for Social Media)

When user provides a video file path, first remove dead zones for fast-paced CapCut/TikTok style:

Detect Silence:

• Run silence detection with aggressive parameters for social media pacing:
  • ffmpeg -i <video-file> -af "silencedetect=n=-42dB:d=0.4" -f null - 2>&1 | grep "silence_" > silence.txt
• Parameters optimized for LinkedIn/Twitter/TikTok:
  • -42dB: Catches natural speech pauses without cutting syllables
  • 0.4s: Fast-paced but not jarring

Parse Silence Timestamps:

• Extract silence start/end times from detection output
• Calculate non-silent segments (the parts to keep)
• Add 0.15s padding before/after each segment to prevent cutting words

Extract and Concatenate Segments:

• Extract each non-silent segment WITH RE-ENCODING to avoid timestamp/keyframe issues:
  • ffmpeg -i <video-file> -ss <start> -to <end> -c:v libx264 -c:a aac segment_N.mp4 -y
  • IMPORTANT: Do NOT use -c copy as it causes video freezing at cut points
• Create concat list file with all segments
• Concatenate WITH RE-ENCODING for smooth playback:
  • ffmpeg -f concat -safe 0 -i segments_list.txt -c:v libx264 -crf 18 -c:a aac <original-name>_edited.mp4 -y
  • IMPORTANT: Do NOT use -c copy for concatenation
• Clean up temporary segment files (segments_list.txt, segment_*.mp4)

Output: <original-name>_edited.mp4 (silence-removed video ready for transcription)

Technical Note: Re-encoding is required (not stream copy) because cutting at arbitrary timestamps creates keyframe misalignment and timestamp discontinuities. While slower, re-encoding ensures smooth playback without freezing.

2. Run Whisper Transcription

Run Whisper on the edited video (with silence already removed):

• Run whisper with: whisper <original-name>_edited.mp4 --model medium --language en --output_format srt
• Monitor command output for completion
• Note the output SRT file location (typically same directory as video with .srt extension)
• Subtitles will already have correct timing for the edited video

3. Review Generated Captions

After whisper completes:

• Read the generated SRT file
• Analyze for common transcription errors:
  • Homophones: their/there/they're, your/you're, its/it's
  • Proper nouns: Names, brands, places often transcribed incorrectly
  • Technical terms: Industry jargon, acronyms, specialized vocabulary
  • Numbers: Dates, times, quantities (written vs numeric)
  • Punctuation: Missing periods, commas, question marks
  • Capitalization: Sentence starts, proper nouns

4. Polish and Output

• Fix identified errors while preserving:
  • Original timing codes
  • Line breaks and caption structure
  • Speaker intent and meaning
• Write polished SRT to new file with -polished.srt suffix
• Report summary of changes made (types of errors fixed, count)

5. Request User Review

• Open the polished SRT file for user to review: open <polished-srt-file>
• Ask user to review the polished SRT file
• Wait for user approval before proceeding
• If user requests changes, make corrections and re-present for approval

6. Generate Thumbnail Caption and Image

After user approves the polished captions:

Generate Caption Text:

• Analyze the video transcript to create 3-4 thumbnail caption options (4-10 words each)
• Each caption should be:
  • A direct quote or paraphrased statement from the video
  • Punchy and attention-grabbing
  • Captures a key insight, surprising fact, or strong opinion
  • Examples: "Why are revenue estimates so bad?", "This changed everything for me", "Nobody talks about this"
• Present caption options to user
• Wait for user to select their preferred caption

Extract and Select Frame:

• Extract sample frames from video at multiple timestamps (1s, 2s, 3s, 5s, 10s):
  • ffmpeg -i <video-file> -ss 00:00:01 -vframes 1 frame-1s.jpg
  • ffmpeg -i <video-file> -ss 00:00:02 -vframes 1 frame-2s.jpg
  • Continue for other timestamps
• If video contains a speaker/face, analyze frames for:
  • Clear, well-lit face
  • Good facial expression (smiling, engaged, not mid-blink or awkward)
  • Good composition and framing
• Select the best frame based on quality assessment

Create Thumbnail with Text Overlay:

• Calculate appropriate font size to prevent text cutoff:
  • Start with fontsize=60 for longer text (7-10 words)
  • Use fontsize=72 for shorter text (4-6 words)
  • Adjust if needed based on video dimensions
• Apply text overlay with careful positioning:
  • ffmpeg -y -i <selected-frame>.jpg -vf "drawtext=text='<approved-caption>':fontfile=/System/Library/Fonts/SFNS.ttf:fontsize=<calculated-size>:fontcolor=white:bordercolor=black:borderw=4:x=(w-text_w)/2:y=(h-text_h)/2" -frames:v 1 <original-name>-thumbnail.jpg
  • Default font: SF Pro (SFNS.ttf) - Apple's system font matching automation.co style
• Output filename: <original-name>-thumbnail.jpg
• Open thumbnail for user review: open <original-name>-thumbnail.jpg
• Show generated thumbnail to user for final approval
• If text appears cut off or thumbnail needs adjustment, regenerate with different sizing

7. Embed Subtitles with FFmpeg

After user approves the thumbnail:

• Use ffmpeg to burn subtitles into the edited video (the one with silence removed)
• Command: ffmpeg -y -i <original-name>_edited.mp4 -vf "subtitles=<polished-srt-file>:force_style='FontName=Arial,FontSize=16,PrimaryColour=&HFFFFFF&,OutlineColour=&H000000&,BorderStyle=3,Outline=2,Shadow=1,MarginV=20'" -c:v libx264 -crf 18 -preset slow -c:a copy <output-file>
• Font size 16 provides good readability without overwhelming the video
• Output filename: <original-name>-subtitled.mp4
• Preserve original video quality with encoding settings
• Report when embedding is complete with output file location

8. Prepend Thumbnail to Video

After subtitle embedding completes:

Create Brief Title Card with Silent Audio:

• Create a 0.1 second (3 frames) video from the thumbnail image with silent audio track:
  • ffmpeg -loop 1 -i <original-name>-thumbnail.jpg -f lavfi -i anullsrc=channel_layout=stereo:sample_rate=48000 -c:v libx264 -t 0.1 -pix_fmt yuv420p -r 30 -c:a aac -shortest title-card-with-audio.mp4
• The silent audio track ensures audio continuity when concatenating
• This duration is imperceptible to viewers but platforms will grab it as preview

Concatenate with Main Video:

• Create concatenation list file:
  • echo "file 'title-card-with-audio.mp4'" > concat-list.txt
  • echo "file '<original-name>-subtitled.mp4'" >> concat-list.txt
• Combine files:
  • ffmpeg -y -f concat -safe 0 -i concat-list.txt -c:v libx264 -crf 18 -preset slow -c:a aac <original-name>-final.mp4
• Clean up temporary files:
  • trash concat-list.txt title-card-with-audio.mp4
• Output filename: <original-name>-final.mp4
• Open final video for user review: open <original-name>-final.mp4
• Report final video location - this is the version ready for X/LinkedIn upload

9. Generate Social Media Content

After embedding completes, analyze the video transcript and generate:

LinkedIn Post/Tweet (very short, max 280 characters):

• Pithy summary of video content
• Hook that captures main value/insight
• Must work as both LinkedIn post and tweet
• Conversational, engaging tone
• No hashtags unless essential

Present the social media post to user, clearly labeled.

Important Notes

• Only fix typos and transcription errors
• Do not rephrase or reword content
• Preserve all timing information exactly
• If uncertain about a correction, note it in the summary for user review
• FFmpeg subtitle embedding creates a new file; original video remains unchanged