multicam-sync

# Multicam Sync Parts 1–3 always run: they produce the synced master, a verifiable fact that every later edit is rebuilt from. Part 4 cuts a draft from it by default; skip it only for an explicit sync-only request. Transcripts do almost all of the work. AI locates; arithmetic decides frames. Use ordinary timelines, tracks, and items only. Never create a compound or nested multicam clip, and never depend on a camera-switcher item. If a main edit already exists, leave it untouched while building the synced master and speaker-follow cutting board on separate timelines. ## Part 1 — Discover the structure Work out what you actually have before aligning anything. Never ask the user how many cameras there are. The material already answers it. ### The two rules that do the work **Same words at the same moment → cannot be sequential spans of one camera.** Overlapping transcripts mean two devices recorded one event simultaneously, so they are different sources or simultaneous angles. **No shared words, and one ends where the next begins → one camera that stopped and restarted.** These are sequential spans of a single capture source, not separate angles. ### Steps 1. Read every asset's transcript, build a pairwise overlap matrix — how much text each pair shares, and where — and split by the rules above. 2. Video assets are **angles**; audio-only assets are **recorders**. 3. Corroborate with filenames, folder structure, camera-model metadata, durations — tie-breakers only, never over transcript evidence. Real shoots ship three cameras all named `C0001.MP4`. ### Report before acting State the structure in the user's terms and wait if anything is ambiguous: > 2 camera angles (Pocket3, 3 spans · Pocket4, 2 spans), 2 audio recorders > (MIC1, MIC2). Session runs 53 minutes. When evidence is thin — a source with little speech, an overlap resting on a handful of matched lines — say so. Do not fill the gap with a guess. ## Part 2 — Align and place One number per clip: timeline time minus source time. Keep that number constant for every piece cut from the clip. Do not time-stretch automatically: a changing offset may be a bad match, a clock difference, or dropped frames, and each needs review. ### Steps 1. **Reference**: the source that covers the whole session with the richest transcript — usually a dedicated audio recorder, not a camera. 2. **Use the renderer when available**: on the separate master timeline, put the untrimmed clips on ordinary source tracks, then call `multicam_sync` with the same-take items and the reference. This is the preferred Web/Desktop path. It uses existing source timing when decisive, otherwise audio correlation; it does not create a multicam object. Accept only `applied`, `already_synced`, or an understood `partial` result. Read `alignmentEvidence` for method, match confidence, correlation, overlap, and relative offset. Read `placementEvidence` for the actual post-sync `timelineSourceOffsetSeconds`. Do not use a skipped or low-confidence item. If the tool is unavailable or finds no confident alignment, leave the master untouched and use the transcript fallback below. 3. **Transcript fallback**: run `scripts/transcript-offset.mjs <utterances.json>` from this skill. Do not improvise the math. It first lets shared phrases vote on a coarse offset, then takes the median of near-identical utterance-pair deltas. It also rejects too few pairs, inconsistent pairs, and early/late drift. Only place files whose output is `confident:true`; report every printed issue for the others. `--help` documents the input and sign convention. 4. **Continuity check** (free, run it): spans of one camera were recorded back-to-back, so span N+1's offset minus span N's must equal span N's duration. Each offset was measured independently, so agreement confirms both. Sub-frame agreement is the norm; a multi-frame gap means dropped frames or a mismatch — say which. **An overlap or gap between placed spans of one camera means an offset is wrong. Recompute it — never trim a span to make it fit.** 5. **Place transcript-fallback results**: shift everything so the earliest clip starts at 0. One video track per camera, one audio track per recorder. Convert seconds → frames once at the end, never accumulating, and **floor at an asset's tail, never round up** — a rounded-up final frame claims source the file doesn't have; renders tolerate it silently, Script editing later refuses it. ### Report Per clip: actual timeline-source offset, method and supporting evidence (renderer confidence/correlation/overlap or transcript pair count/spread/drift), and the continuity-check result. ## Part 3 — Identify and label Turn "track 1 / track 2" into who is actually on it. Everything here is evidence-first: a wrong confident claim is a failure; an honest "can't tell" is not. ### Who is in the session Speakers usually name themselves or each other — self-introductions, banter. Pull real names from the transcript. If none appear, A/B is fine; never invent names. ### Which camera frames whom From the transcript, pick 3–4 moments where only one person is talking, spread across the session. View the synced frame from **every** camera at the same wall-clock moment: the talking face identifies the person; clothing and seating anchor identity across angles. Two-shots are identity anchors, not noise. Classify framing while you're there: close-up / medium / two-shot; note reframing if the dominant framing changes. Angle labels are summaries, not promises: framing can drift, fail, then recover. Inspect the exact synced interval before every cut; never choose from the label alone. Hard check before concluding: at each self-introduction, the face whose lips are moving is that name's owner. "Both cameras frame the same person" contradicts a two-camera, two-mic, two-voice structure — treat that conclusion as an error until frames at both introductions prove it. ### Which mic belongs to whom A lav is dramatically louder for its wearer — typically 15 dB or more. Measure, don't infer: - During one person's solo speech vs the other's, compare **the same track against itself**. The in-track contrast cancels recorder gain. Above ~4 dB it decides; below, say "indistinct" — that itself is a finding (ambient mic, shared mic). - Judge the **distribution**, not one moment: consistent → owned mic; 50/50 → not a personal mic; flips mid-session → handheld passed around or seats changed. - **Never use utterance counts or transcript volume as evidence of mic ownership.** Crosstalk transcribes fine; both speakers appear fully on both mics' transcripts. The overall loudness difference between two mics is not evidence either — only the in-track solo-vs-solo contrast is. - Fragmented diarization heals mechanically: cluster speaker-ids by their median cross-track level difference. Never hand-reconcile speaker ids across assets — they are per-asset serials. ### Which source is the program audio Decide what the viewer will hear. Default: dedicated recorders beat camera embedded audio. The user's word beats everything — "camera A has the good audio" is a program audio assignment; that camera's audio track then behaves exactly like a recorder (its offset is already known from Part 2). Record the assignment in the report. With no clean recorder, compare camera mics at the same solo-speech moments. Prefer one continuous source that is good enough; switch for a speaker or passage only when another is clearly better for a sustained stretch and the handoff is inaudible. Judge intelligibility, noise, clipping, and reverb — not labels or utterance counts. ### Label Rename tracks in place with the shortest evidenced label: `Cam · <subject or view>` and `Audio · <speaker or source>`. Append a shot size only when it distinguishes otherwise similar angles: `WS` (wide shot), `MS` (medium shot), or `CU` (close-up), for example `Cam · Speaker A · MS` or `Cam · Two-shot · WS`. Otherwise omit it. Fall back to numbered labels when identity is uncertain. Deliver an evidence table alongside: claim | evidence (timestamp + what was seen or measured) | confidence. Decline to label what the evidence doesn't support — fragmented transcript speaker ids are usually in that category — and say so. ### Deliver the master, then cut by default The master is the deliverable — report structure, offsets, identities, and evidence. Note that it stacks angles and is a reference, not something to watch: the top track covers the others. If the user explicitly asked for sync only, stop here. Otherwise continue directly to Part 4; do not wait for a separate cut-to-speaker request. ## Part 4 — Cut to the speaker (default unless sync-only) Turn the synced master into a watchable draft: one video track that follows the conversation, program audio continuous underneath. ### The master is never edited All cutting happens on a **new timeline** (same fps/canvas). The synced master is the source of truth every derived cut can be rebuilt from — if it changes, every offset becomes unverifiable. When any instruction, taken literally, would break sync (e.g. "start the audio at frame 0" when its synced position is not 0), keep sync and say why. Preserving sync outranks literal wording. Treat the speaker-follow timeline as a source or cutting board, not as an opaque clip. When the user wants part of it in an existing edit, materialize only those ordinary picture and audio ranges into the main timeline. To substitute another angle later, take the same wall-clock interval from that camera using the master offset. The master is the timing record; no hidden multicam data structure is required. ### The conversation drives the cut Build one conversation document first: each person's speech taken from their assigned program source (Part 3), crosstalk dropped, interleaved by wall-clock via the Part 2 offsets, with real names and timestamps. Then write the angle plan — start–end, angle, reason — **before placing anything**. If the user also asks to remove fillers, shorten answers, or restructure the conversation, use the talking-head workflow to decide which speech ranges remain. Multicam owns sync and angle choice; talking-head owns content. Finalize the content plan before materializing picture cuts. ### Editorial objective Keep the viewer oriented, emotionally informed, and visually awake with the fewest cuts that add something. **Meaning chooses what to show. Rhythm chooses when to cut. Orientation chooses how wide to go.** Stay while the frame is still revealing; move when another frame gives more; return to the room when the relationship needs refreshing. Angle rules (defaults, user's brief wins): - **Carrier of the moment**: the active speaker is the baseline, not the law. Show the speaker when information originates there, the listener when the reaction is the meaning, a pair or subgroup when the relationship carries the beat, and the room when attention is divided. - **Shot scale**: use the smallest grouping that preserves the beat. `CU` isolates thought or emotion; `MS` is the conversational default; a two-shot or group shot shows a relationship; `WS` restores geography. Use a wider view at a new question or topic, a participant change, overlap, shared laughter or silence, physical movement, or after a long run of isolated singles. Hold it long enough to read — usually 2–5s — then tighten when attention concentrates. - **Duration pressure**: a normal shot needs about 2–3s to arrive. Treat 4–10s as a useful conversational range, not a metronome. After roughly 8–12s on an unchanged single, actively look for a motivated alternative. By 20–25s the hold should be deliberate. There is no maximum while performance, emotion, or visual information is still developing. - **Speaker changes**: do not chase every sound. A <2s interjection usually stays on the current shot. Let an unanticipated new speaker begin for about a second before cutting; a direct question may motivate showing the respondent while they prepare to answer. A short important line — introduction, direct address, punchline — earns a shot; widen around it if necessary. - **Reactions**: use a visible, truthful reaction when it changes how the line lands or refreshes a static hold. A reaction is usually 1.5–3s. Do not insert a generic nod merely for variety, and never borrow a reaction from another wall-clock moment. - **Rhythm**: cut on a thought, breath, gesture, look, laugh, or relationship change, not on a timer or arbitrary word boundary. Sentence boundaries are safe, not mandatory. - **Seams**: one clean speech seam at a sentence boundary needs nothing. Cover a cluster of visible jump cuts with one wall-clock-synced reaction or relationship shot spanning the cluster. Never break source sync to manufacture coverage. - A camera restart under an unchanged angle is a same-angle join, not an editorial cut. Count and report it separately. ### Materialize flat - **One video track**, alternating angle segments, no gaps. Each segment's source offset comes from the master placement math, never re-derived from transcripts. - For a full-length speaker-follow draft, keep program audio continuous and untrimmed at its synced position. For a shortened or reordered cut, use identical kept source-time ranges on every program mic so picture and audio cannot drift. - Mute every camera segment's embedded audio. - If audio outruns picture (recorders stopped later), keep the audio and leave the tail dark. Never fabricate or freeze picture to cover it. ### Keep program audio coherent Keep every isolated program mic open across each retained range. When program audio comes from camera mics, use only the chosen source for that passage, crossfade at a quiet boundary, and do not stack them. Word timestamps carry ±30–60ms of ASR noise, so boundaries need natural handles; hard cuts on word timestamps clip breaths and word onsets. Backchannel ("嗯", laughter) on an idle mic is part of the conversation. After content editing, run the talking-head workflow's audio smoothing step. If mic isolation is weak, flag it for an audio pass rather than gating speakers independently. ### Verify last, and prove it with numbers Verification is the **final action**, after every edit and every visual check. Anything that touches the timeline — including dragging in a browser to look at a frame — can move an item. **Inspection is not read-only.** If you interacted with the editor UI at any point, re-run these checks afterwards; a pass from before that interaction is void. - **Master wall-clock invariant**: on the synced master, timeline time − source time must equal that source's master offset. - **Derived-cut sync invariant**: for each retained passage, compute `cut time − (source time + master offset)` for the picture and every program audio source at the same cut time. Their values must agree within a frame. A shortened or reordered edit can have a different displacement for each passage; do not force it back to the master's offset. - **Angle spot check**: at a few sampled shots — include the longest — the dominant speaker in the window must match the angle's person. Report the **actual values**, not the word "verified": `master MIC1/MIC2 offsets 3.700 / 3.700; cut passage at 111, picture/audio master times 123.000 / 123.000`. A claim you cannot print numbers for has not been checked. The same rule governs recovery. If you disturb an item, **restore every field and re-read the row to prove it** — position, duration, and source offset each fail independently, and fixing the one you noticed is not a restore. Also report: cut count, shot length min/avg/max, the conversation document, the plan, and every place a rule conflicted with the material plus what you chose.