fframes-video

# Making videos with fframes [fframes](https://github.com/dmtrKovalenko/fframes) is a Rust library that renders video from code. A video is a Rust struct with a `render_frame(frame) -> Svgr` method: for every frame it returns an SVG tree, built with the `svgr!` macro (SVG markup with `{rust expressions}`). Scenes split the timeline, `timeline!` and springs animate values, an audio map places sound, and ffmpeg encodes the result. Each project gets a command line (from `fframes::cli`). Use it for everything: render a video, open the real-time preview window, and look at frames, contact sheets and loudness numbers as files you can read. Read `references/design.md` before designing, `references/api.md` while writing code and `references/audio.md` for sound. When matching a supplied video, first follow the reference-analysis guidance in `references/design.md`. Its cuts, pacing and motion take precedence over the design defaults below; fast edits and intentional black frames may be part of the requested result. ## 1. Install Rust from <https://rustup.rs>, then the system libraries ffmpeg is built with: ```bash # macOS brew install pkg-config ffmpeg x264 x265 opus nasm ninja # Debian / Ubuntu sudo apt-get install -y yasm nasm ffmpeg libx264-dev libx265-dev libopus-dev libclang-dev clang ninja-build libvpx-dev libasound2-dev # Arch sudo pacman -S ninja yasm nasm ffmpeg x264 x265 opus clang ``` Windows links a prebuilt FFmpeg 9 shared build instead (set `FFMPEG_DIR` to the unzipped `ffmpeg-n9.0-latest-win64-gpl-shared` build from BtbN/FFmpeg-Builds, add its `bin` to `PATH`, install LLVM with `winget install LLVM.LLVM` and set `LIBCLANG_PATH`). Details are in the fframes README. ## 2. Create a project ```bash curl -fsSL https://raw.githubusercontent.com/dmtrKovalenko/fframes/main/scripts/new-video.sh | bash -s -- my-video --yes # or install the generator once and use it directly: cargo install --locked cargo-fframes cargo fframes new my-video --format landscape --fps 30 --yes ``` Options: `--template single-scene|multi-scene`, `--format landscape|portrait|square|uhd`, `--fps`, `--title`, `--backend`, `--dir`. The project depends on the fframes release that matches the installed `cargo-fframes`; `--git` uses the repository's `main` branch instead. Always pass `--yes` so nothing waits for input. Templates: - `single-scene` (default): one scene. It adapts to every format; start here for single-shot clips, title cards and portrait video. - `multi-scene`: two scenes. Start here for anything with several scenes; it only accepts `landscape` and `uhd`. The template content is placeholder code that shows the API. Replace its layout, colors, fonts and decorations with a design made for the video the user asked for. **Use the Skia GPU backend (the default).** `cargo fframes new` picks Skia on Metal (macOS) or Vulkan (Linux, Windows). It renders about 10x faster than the CPU backend and gives you the real-time `preview` window. On macOS and Linux the first build downloads prebuilt Skia and ffmpeg and only compiles the Rust dependencies (about a minute; other targets, or `metal` and `vulkan` together, compile Skia from source for ~20 minutes). Start it right away and write the video while it runs: ```bash cd my-video && cargo build --release # run in the background, the first build is the slow one ``` `--backend cpu` needs no Skia but has no preview window; pick it only when there is no GPU. The project renders as generated: ``` my-video/ src/lib.rs # the video (from the template) src/main.rs # the command line media/ # fonts, images and audio compiled into the binary (one starter font) README.md # the commands below ``` Every command below is `cargo run --release -- <command>`. Define a short shell function (a variable like `R="cargo run --release --"` does not split into words in zsh, the macOS shell): ```bash R() { cargo run --release -- "$@"; } ``` ## 3. The loop **Do not write tests at all when using this skill.** Do not create unit, integration, snapshot or visual-regression tests, test modules, or test harnesses. Validate the video with `inspect`, rendered frames, contact sheets, audio checks and the native preview. **For the GPU backend, offer the native preview before a full render.** Do not render the whole video as a routine review step. Once the changes are ready to review: 1. Show a complete command in a copyable shell code block, using the actual project path and package name; do not require the user to have defined the `$R` helper. For example: `cd /absolute/path/to/fframes && cargo run --release -p my-video -- preview 0s`. 2. Explicitly offer to run that preview for the user. If they have already authorized opening or refreshing the preview, show the command and continue within that authorization. After a requested change, set the command's start time to a specific second at or just before the changed section, with enough lead-in to see its entrance or transition. For example, `preview 8.5s` lets the user review a change around 9 seconds without replaying the beginning. State which section they will see. Render the full video when the user requests the final export; do not require another confirmation if that request is already explicit. After every change: 1. `$R timeline` lists the scenes with their frame and second ranges, and every audio track. Check structure and pacing here before looking at pixels. 2. `$R inspect` checks a frame every 0.25 s plus the first and last frame of every scene. It reports missing images, fonts or glyphs, text cut off by the edge of the canvas, invalid SVG (zero-sized rectangles, bad radii), broken transforms and panics, each with its time and scene. It exits with code 2 on errors. Fix everything it reports. A warning that only appears on a scene's first frames is usually an entrance; check it with a strip. 3. `$R strip <scene or range> -n 12` writes a contact sheet (`strip.png`): evenly spaced frames in one labelled image. Open it with your image tool. It is the fastest way to judge layout, rhythm and motion. 4. `$R frame Intro@end,Outro@50%` writes full-size PNGs into `frames/` for detail checks (typography, alignment, contrast) and prints problems found in those frames. 5. `$R onion "Intro@0..Intro@1s" -n 6` blends frames into one image (`onion.png`) to show the path and spacing of a movement: easing, overshoot, stagger. 6. `$R preview Intro` opens a real-time window with sound for the user to watch (space play/pause, h/l seek a second, j/k step a frame, q quit). It blocks until closed, so start it in the background when running it for the user. Show the command and offer to launch it as described above; after edits, use a specific second near the change. When an encoded sample is needed, `$R render Intro --draft` encodes one scene at half resolution. `$R render` writes the full video for a requested final export. Rules: - Look at the PNGs before saying anything looks good. - Prefer `strip` to many `frame` calls; add `--scale 0.5` when composition is all you need. - Add `--json` when parsing output: the result goes to stdout, progress to stderr. - Keep `--release`: debug builds render many times slower. ### Addressing time `120` (frame), `3.2s`, `500ms`, `1:05.5`, `50%`, `start`, `end`; scenes by struct name, `Intro` (also matches `IntroScene`, case-insensitive), `#3` (scene index), `Intro[1]` (second scene of that type); inside a scene `Intro@1.2s`, `Intro@12`, `Intro@50%`, `Intro@end`. Ranges: `a..b` (end exclusive), `a..`, `..b`, `all`, or a scene name for the whole scene. Separate several times with commas. ### All commands | command | use it to | | --- | --- | | `timeline` | see scenes, durations, audio tracks and their mix settings | | `inspect [RANGE] [--every 0.1s \| --all-frames] [--fail-on warning]` | find problems without rendering pixels | | `strip [RANGE] -n N [--columns 4] [--width 480]` | review flow and motion in one image | | `frame TIMES [-o dir] [--svg]` | full-size PNGs (and the laid-out SVG) of chosen frames | | `onion RANGE -n N` | see a movement's trajectory and easing | | `svg TIME` | read a frame's final SVG as text (positions, text, colors) | | `preview [TIME] [--paused] [--mute]` | real-time window with sound, for the user | | `render [RANGE] [-o out.mp4] [--draft]` | encode the video or a part of it (audio cut to match) | | `snapshot TIMES [--update]` | compare frames with approved PNGs, `.diff.png` marks changes | | `audio analyze [RANGE] [--waveform w.png]` | loudness (LUFS), true peak, clipping, silence, per scene | | `audio at TIMES` | which sounds play at a moment, where in their file and how loud | | `audio render [RANGE] -o a.wav` | the mix as a WAV file | ### Browser editor fframes also has a web editor with a timeline and scrubbing, useful when a person wants to tweak a video interactively. It runs the video compiled to WebAssembly and needs Node.js plus `wasm-pack`; the setup is the `editor/` folder of the hello-world example in the fframes repository (<https://github.com/dmtrKovalenko/fframes/tree/main/examples/hello-world>). Mention it when the user asks for a GUI editor. For watching use `preview`, and for your own review use the CLI. ## 4. Writing the video ```rust impl Video for MyVideo<'_> { const FPS: usize = 30; const WIDTH: usize = 1920; const HEIGHT: usize = 1080; fn duration(&self) -> Duration<'_> { Duration::Auto } // sum of the scenes fn audio(&self) -> AudioMap<'_> { AudioMap::none() } // see references/audio.md fn define_scenes(&self) -> Scenes<'_> { Scenes::from(vec![&self.intro as &dyn Scene, &self.main]) } fn render_frame<'a>(&'a self, frame: Frame, ctx: &FFramesContext<'a, '_>) -> Svgr<'a> { fframes::svgr!(<svg xmlns="http://www.w3.org/2000/svg" width={Self::WIDTH} height={Self::HEIGHT}> <rect width={Self::WIDTH} height={Self::HEIGHT} fill={BACKGROUND} /> {ctx.render_scenes(&frame)} </svg>) } } ``` - One scene per idea, 2-6 s each. Inside a scene `frame.seconds()` counts from the scene start. - Default to `frame.animate(fframes::timeline!(...))` directly in `svgr!` attributes for motion with fixed values. `svgr!` lifts these animations into cached statics. Define easings inline in the timeline, including `Easing::Spring { mass: 1.0, stiffness: 180.0, damping: 20.0 }`; named easing constants are optional. Leave the end time off a spring so it settles on its own. - For dynamic values or calls outside that optimized pattern (including `animate_loop`), construct timelines once on `self` and sample them by reference. Use cached `AnimationRuntime` values for starts that change after construction or custom/subframe clocks. Camera parameters and shader uniforms can use these APIs too; do not duplicate springs or tween interpolation. See `references/design.md` and `references/api.md`. - Markup without `{}` is cached across frames. Keep decoration literal and put animated values on a wrapping `<g transform={..} opacity={..}>`. - `render_frame` runs for every frame on several threads: no panics, file reads or heavy work in it. Prepare data in the constructor or a `OnceLock`. Media lookups return `Option`; fall back to `Svgr::empty()` instead of `expect`. - Put every font file in `media/` and refer to it by family name, with a numeric `font-weight`. - Measure text rather than guess: `frame.text_width`, `frame.text_fit(.., TextOverflow::Ellipsis)`, `frame.text_break_lines` for paragraphs. `inspect` catches text leaving the canvas, not text leaving its own box, so check boxes in a `frame` PNG. - GPU shaders (SkSL or Shadertoy GLSL) run on the Skia backend through `fframes::Shader`; they take images and synced video frames as input (green-screen keying, color grading). See `references/api.md`. ## 5. Making it look good The short version of `references/design.md`: - One idea per scene, large type, margins of 8-10% of the width, two or three colors plus neutrals and one accent for emphasis. - Elements enter with a spring or ease-out over 300-600 ms and leave faster, with ease-in over 200-300 ms. Related items stagger by 60-120 ms. Give the viewer 1-2 s to read after the motion settles, and never move everything at once. - Cross-fade scenes (`fn overlap(&self) -> Overlap { Overlap::Previous(0.4) }`; the overlap is added to the scene's `duration()`, see `references/api.md`) or carry an element across the cut. Keep a little motion during holds so they do not look frozen. - At 1920x1080: titles 96-140 px, body 44-60 px, at most about 8 words per line and 3 lines per card. Portrait 1080x1920 is watched on a phone: same pixel sizes or larger, content inside the middle 80% because platform UI covers the top and bottom. - Check every scene against the checklist in `references/design.md` with a strip. ## 6. Sound Put audio files in `media/` (compiled in, mono) or load a folder at runtime with `MediaDirectory` for stereo, then place them: ```rust AudioMap::from([ AudioTrack::new("music.mp3", Second(0.)..Eof).gain_db(-18.).fade_in(1.).fade_out(2.).duck_under_voice(), AudioTrack::new("vo.wav", Second(0.6)..Eof).voice(), AudioTrack::new("whoosh.wav", Second(3.1)..Eof).gain_db(-8.), ]) ``` Time sound effects from the same constants that drive the animation. Check levels with `$R audio analyze --waveform w.png` (about -14 LUFS integrated for web video, true peak below -1 dBTP, no unintended silence) and `$R audio at 3.1s` for what plays at an event. Then let the user listen in `$R preview`. ## 7. Finish 1. `$R inspect --fail-on warning` passes, or the remaining warnings are understood entrances. 2. A strip of every scene looks right and key frames are checked at full size. 3. `$R audio analyze` shows sensible levels. 4. For GPU work awaiting review, show the preview command and offer to run it at the relevant second. When the user requests the final export, run `$R render -o out.mp4`, then confirm size, frame count and audio with `ffprobe -v error -show_entries stream=codec_type,width,height,nb_frames,duration out.mp4`. For preview review, provide the concrete preview command, its start time and the offer to launch it. After final export, provide the output path and duration. ## Troubleshooting - Build fails in `ffmpeg-sys-fframes`: a system library from step 1 is missing (`nasm`, `pkg-config`, the codec `-dev` packages). - Build fails in the Skia bindings (`skia-bindings`) with bindgen or libclang errors (Skia is only compiled from source when no prebuilt matches, e.g. `metal` and `vulkan` together): point `LIBCLANG_PATH` at a working libclang (on macOS Xcode's: `export LIBCLANG_PATH=$(xcode-select -p)/Toolchains/XcodeDefault.xctoolchain/usr/lib`). - Skia bindings fail to compile on macOS 27 / Xcode 27 with ``cannot find type `_Traits` `` in `std___hash_table___node_allocator`: only happens when Skia is compiled from source (the prebuilt binaries ship their bindings). `skia-bindings` 0.153.3 lacks the bindgen rule (rust-skia [#1335](https://github.com/rust-skia/rust-skia/pull/1335)); use one GPU backend so the prebuilt download matches, or patch a copy of `skia-bindings-0.153.3` (add `"std::__hash_table.*",` after `"std::__tree.*",` in `OPAQUE_TYPES` in `build_support/skia_bindgen.rs`) through `[patch.crates-io]`. - ``can't find crate for `fframes_media_dir_macro` `` on macOS 27 while the file exists: the proc macro was linked by an older Rust whose output the macOS 27 loader rejects (a "LINKEDIT string pool" error). Update Rust (`rustup update`; 1.98 works) and rebuild. - Text renders in the wrong font or not at all: `inspect` shows "No match for ... font-family"; add the font file to `media/` and use its exact family name.