tutorial-videos

# Tutorial Video Workbench Architecture and rationale live in `tools/tutorial_videos/README.md` — read it first. This skill is the operational runbook. ## Build videos ```sh make tutorial_video TUTORIAL_LOCALE=de # one locale (~5-8 min) make tutorial_video TUTORIAL_LOCALE=de TUTORIAL_DEVICE=mobile # phone-shaped window, `_mobile`-suffixed output make tutorial_videos_all # all TUTORIAL_LOCALES ``` `TUTORIAL_DEVICE` (`desktop`|`mobile`, default `desktop`) picks the capture window size — see the README's "Desktop vs. mobile" section for the breakpoint rationale and the `tutorialDeviceIsMobile()` test hook. Preconditions (fail fast if missing): - `.env` at repo root with `GEMINI_API_KEY`, `MELIOUS_API_KEY`, `MELIOUS_BASE_URL`. If absent, the keys can be extracted from the dev app's `ai_config.sqlite` provider rows (never print or commit them). - Sibling `../OpenMontage` checkout at the commit pinned in `tools/tutorial_videos/config/openmontage.pin`, bootstrapped via `make -C ../OpenMontage setup`. - `Xvfb`, `ffmpeg`, `pactl`, `paplay`, `x11-utils` installed; Flutter via `fvm`. Output lands in `build/tutorial_videos/<scenario>_<locale>.mp4`. Verify with frame extraction (`ffmpeg -ss <t> -frames:v 1`) and check: cursor visible and gliding, HUD clock top-center, transcript/proposals on screen, narration audible at step starts (`volumedetect`), duration ≈ warped timeline total. ## Publish to R2 ```sh make tutorial_video_publish TUTORIAL_SCENARIO=create_task_from_audio TUTORIAL_LOCALE=de make tutorial_video_publish TUTORIAL_SCENARIO=create_task_from_audio TUTORIAL_LOCALE=de TUTORIAL_DEVICE=mobile ``` Uploads `build/tutorial_videos/<scenario>_<locale>.mp4` (or, with `TUTORIAL_DEVICE=mobile`, `<scenario>_<locale>_mobile.mp4`) to Cloudflare R2 at the matching `tutorial-videos/...` key and prints its public r2.dev URL — this is the URL the docs-site `TutorialVideo` component builds from `TUTORIAL_VIDEO_BASE_URL`/`tutorialVideoBaseUrl`. `TUTORIAL_DEVICE` must match whatever `--device` the video was built with, or publish will look for a file that was never produced. Full one-time Cloudflare setup (API token, enabling the bucket's public r2.dev URL) and the `.env` variable list are in the README's "Publishing to Cloudflare R2" section — read it before the first publish. `boto3` (needed only by publish, not by build/tts/validate) is not installed into the system Python on this host (PEP 668). Either run publish through `tools/tutorial_videos/.venv` (`.venv/bin/python3 -m tutorial_videos publish --scenario ... --locale ...`, created via `python3 -m venv .venv && .venv/bin/pip install boto3 pyyaml`), or install `boto3` into whatever Python `make tutorial_video_publish` resolves to. ## The pipeline in one paragraph `python3 -m tutorial_videos build` (run from `tools/tutorial_videos/`) does: Gemini-TTS pre-pass (cached by content hash → durations manifest) → Xvfb + virtual-mic null sink + ffmpeg x11grab capture → `fvm flutter drive` runs `integration_test/tutorial/<scenario>_tutorial_test.dart` against the real app (penguin demo world + Melious/Voxtral/Qwen agent config seeded, UI language via `MANUAL_LANGUAGE` settings row) → the test emits `timeline.json` (step boundaries, wait spans, zero epoch) → `timewarp.py` plans fast-forward segments for the waits (narration never overlaps) → `compose.py` warps the capture and drives the pinned OpenMontage headlessly (narration placement + mux) → ffprobe validation. ## Debugging a failing run 1. The run log tail shows the failing step and a `Diagnostics:` line (current route, detail stack, key widget presence). 2. **Look at the failure screenshot first**: `build/tutorial_videos/failure_<context>.png` is captured at the moment of every timeout — it almost always explains the failure instantly. 3. Reproduce faster without video: run the same test via `flutter drive` directly (see the doc comment in `integration_test/tutorial/tutorial_harness.dart` for the env contract); Xvfb + sink setup must match `session.py`. 4. Common causes, in order of historical frequency: a finder matching an offstage duplicate (scope to the visible page + `.hitTestable()`), a scroll helper given a non-vertical scrollable, a modal/dialog blocking the UI (onboarding, discard-recording), the agent's `automaticUpdatesEnabled` flag not sticking (set-and-verify), cloud latency beyond a `pumpUntil` timeout. ## Adding a locale 1. Add the locale's text to `tools/tutorial_videos/config/scenarios/<scenario>.yaml`: `title`, `dictionary`, every step's `narration`, the dictation step's `dictation_text`. Write informal register (du/tu) per repo l10n rules; keep the penguin vocabulary. 2. Add per-stream `style` lines for the locale in `config/voices.yaml` (style prompts in the target language — Voxtral/Gemini behave better). 3. `python3 -m tutorial_videos validate --scenario <s> --locale <l>` must pass, then build. The app UI localizes itself via `LOTTI_MANUAL_LOCALE` (same mechanism as manual screenshots; for a wholly new app locale, run the `add-flutter-docusaurus-locale` skill first). 4. Localized finder strings in the test (`localized(en:, de:)` / `manualScreenshotText`) need the new locale only if its UI labels are used as finders — prefer keys over text finders where possible. ## Adding a scenario 1. New YAML in `config/scenarios/<name>.yaml` — schema documented in `create_task_from_audio.yaml` (unique step ids, exactly one `dictation: true` step, per-locale text, `min_duration` floors). 2. New `integration_test/tutorial/<name>_tutorial_test.dart` built on `TutorialAppHarness` + `TutorialDriver`. Copy the structure of `create_task_from_audio_tutorial_test.dart`: - one `driver.step(id, action)` per YAML step, ids matching exactly; - interactions via `driver.tapLikeUser` (cursor glide), waits via `driver.pumpUntil*` (records fast-forwardable wait spans), scrolling via `driver.scrollIntoView` with a **vertical** scrollable finder; - anything that must be SEEN gets asserted via visible widgets, not just the DB; - wire the `diagnostics` and `onTimeout` hooks for failure screenshots. 3. Iterate with builds; expect the first runs to fail — read the failure screenshots, they carry the answer. ## Non-negotiable gotchas The full list with rationale is in the README's "Hard-won constraints" — the ones that bite hardest: `flutter drive` not `flutter test` (no window otherwise); `GDK_BACKEND=x11` + empty `WAYLAND_DISPLAY` (or the app opens on the real desktop); Voxtral prompts in the audio's language (else it translates); `automaticUpdatesEnabled` set-and-verify (initial wake clobbers early writes); 16ms pump cadence during holds (else animations render choppy); timeline starts after startup settles; assert transcript keywords, not exact strings. ## Adapting / extending - **App Store preview**: `config/scenarios/app_store_preview.yaml` is the narration of `make store_preview_ios`, not a tutorial — never `make tutorial_video` it. Its step ids must stay the `_beatIds` of `integration_test/store_preview_test.dart` (`tests/test_app_preview.py` guards it), and all four lines together must leave the beats under 30 s (`python3 -m tutorial_videos.app_preview pacing` fails first if not). See the README's "The App Store preview reuses the narration". - **Compositor changes**: all OpenMontage API usage is isolated in `tutorial_videos/om_compose_driver.py`; bump the pin in `config/openmontage.pin` deliberately and re-verify determinism (compose twice, compare `ffmpeg -f framemd5` of the streams — never file hashes). - **TTS vendor**: implement the `TtsEngine` protocol (`tts/base.py`) next to `tts/gemini.py`; voices/styles per locale in `config/voices.yaml`. - **Speed/pacing knobs**: `timewarp.py` (`MAX_SPEED`, lead-in, narration gap); per-step floors in the scenario YAML. - **Character overlay (future)**: render transparent PNG frames via the character film-strip harness and composite as an extra layer keyed to `timeline.json` step ids.