rerun-mp4

# Rerun mp4 ingestion `Mp4Reader` turns one `.mp4` file into a lazy chunk stream on **one entity**: compressed video samples, no decode to pixels, nothing re-encoded unless it has to be. Everything is configured on the constructor; `stream()` takes no arguments. Stream mechanics after `.stream()` (filter, map, merge, collect, write) are in `rerun-chunk-processing`. ## The API ```python from rerun.experimental import Mp4Reader, Mp4TranscodeOptions reader = Mp4Reader(video_path, entity_path="/camera/front") # mode="stream" by default stream = reader.stream() # lazy: nothing is decoded yet ``` Every parameter after `path` is keyword-only. One reader handles one file — for several cameras, build one reader per file and `LazyChunkStream.merge(...)` them (see below). ## Two modes | `mode` | emits | when | | -------------------- | ------------------------------------------------------- | ----------------------------------------------------------------- | | `"stream"` (default) | a codec chunk, per-GOP sample chunks, a keyframe marker | almost always — every frame is time-indexed and queryable | | `"asset"` | the whole file as one blob, plus a frame index | the codec cannot be a `VideoStream`, or the source starts mid-GOP | Reach for stream mode; the next section covers exactly what it emits. Asset mode copies the entire file into the recording as one blob, so nothing downstream can look at a single frame without the whole asset. It is the fallback for codecs a `VideoStream` cannot carry (`mp4v`, image-sequence mp4), and it is what `rerun video.mp4` and the built-in file importer use. ## What stream mode emits For `tests/assets/video/Big_Buck_Bunny_1080_1s_h264_nobframes.mp4` (30 frames, one GOP) with `entity_path="/camera/front"`, the whole output is three chunks: ``` [0] entity=/camera/front static=True rows=1 timelines=[] cols=['VideoStream:codec'] [1] entity=/camera/front static=False rows=30 timelines=['video'] cols=['VideoStream:sample'] [2] entity=/camera/front static=False rows=1 timelines=['video'] cols=['VideoStream:is_keyframe'] ``` - **One static chunk, one row**, holding `VideoStream:codec`. It carries **no timeline** — code that walks the stream must handle that (`if chunk.is_static: continue`). - **One temporal chunk per GOP**: a keyframe plus every sample that depends on it, up to (not including) the next keyframe. Samples only — the keyframe flags are not a column here. - **One trailing `is_keyframe` marker chunk**, holding a *sparse* `True` row at each keyframe's time and nothing else. Keeping it out of the sample chunks is what lets a keyframe-only query skip the sample payload, and what `collect(optimize=…)` accepts as canonical. It is list-per-row, so a row reads back as `[True]`, not `True`. - **Timeline `video`**, `duration[ns]`, values are the mp4 PTS from the start of the video. - **Every chunk on the same entity.** `entity_path=None` (the default) derives it from the file's *absolute* path, so `foo/video.mp4` run from `/data` becomes `/data/foo/video.mp4`. Pass `entity_path` explicitly in any real pipeline. A short clip is often a single GOP, so "three chunks total" is the normal shape — not a sign that something was dropped. `chunk_by_gop=False` emits one Rerun chunk per sample instead (the marker chunk is unaffected). That is a debugging shape — one chunk per frame is a poor storage layout, and it also makes a following optimize pass **6× more expensive** (measured below) — so leave the default alone unless you are inspecting individual samples. Distinguish the three by their columns, not by position or `is_static`: the codec chunk is the static one, sample chunks carry `VideoStream:sample`, and the marker chunk carries only `VideoStream:is_keyframe`. Supported stream-mode codecs are exactly the five `VideoCodec` values: H264, H265, AV1, VP8, VP9. ## Transcoding through FFmpeg `VideoStream` cannot yet model DTS != PTS, so an H.264/H.265 source with container-level B-frame reordering **cannot be emitted directly**. The reader handles that itself: it re-encodes through FFmpeg with `-bf 0`, streams the result back as a fragmented mp4, and turns each fragment into one GOP chunk, so only one GOP is resident at a time. This is automatic and invisible in the API — the output has the same shape as a clean source. It does need an `ffmpeg` executable. Asset mode is unaffected by B-frames. `Mp4TranscodeOptions` (stream mode only) additionally *requests* a transcode: | field | effect | | ----------------- | -------------------------------------------------------------------------- | | `output_codec` | re-encode to another `VideoCodec`; the emitted `VideoStream:codec` follows | | `gop_size` | force a keyframe every N frames — the knob for seek cost in the viewer | | `try_gpu` | best-effort hardware encode, NVENC / VideoToolbox only | | `ffmpeg_override` | use this `ffmpeg` instead of the one on `PATH` | Requesting the `output_codec` the source already uses is a **no-op**: it stays on the direct, no-FFmpeg path. With `chunk_by_gop=True`, `gop_size=N` makes every GOP chunk but the last hold exactly N samples — a 30-frame clip at `gop_size=10` gives 10, 10, 10. `try_gpu` realistically covers H264/H265 plus AV1 on newer NVIDIA; VP8/VP9 always fall back to software, and it does nothing unless a transcode is already happening. ```python Mp4Reader( video_path, entity_path="/camera/front", # ~1s GOPs on a 60fps source, so seeking in the viewer stays snappy. transcode=Mp4TranscodeOptions(gop_size=64), ).stream() ``` ## Timelines: mp4 PTS is not your recording's clock The emitted times are always PTS — elapsed nanoseconds from the start of *that video*. In a multi-sensor recording, that is almost never the timeline you want to align on. - `timeline_name="real_time"` renames the timeline. - `timeline_type="timestamp"` only **retypes** the same PTS values as nanoseconds since the Unix epoch. The reader does not shift them, so on its own it renders the video near 1970. It is meaningful only paired with a retag step. - **Retag with `stream.map(...)`.** Samples arrive in presentation order (B-frames are already stripped), so a running cursor maps sample `i` to `capture_times_ns[i]` — but the cursor must skip the keyframe marker chunk, which holds no samples. Map that one through the PTS the samples were already assigned: ```python SAMPLE_COL = "VideoStream:sample" def _reindex_to_capture_times(stream, capture_times_ns, timeline_name): cursor = 0 pts_to_time = {} def _retag(chunk): nonlocal cursor if chunk.is_static: # the codec chunk carries no timeline return chunk batch = chunk.to_record_batch() col_index = batch.schema.get_field_index(timeline_name) old_field = batch.schema.field(col_index) old_pts = np.asarray(batch.column(col_index).cast(pa.int64())) if SAMPLE_COL in batch.schema.names: # Clamp in case the decoder yields a slightly different frame count. indices = np.clip(np.arange(cursor, cursor + chunk.num_rows), 0, len(capture_times_ns) - 1) cursor += chunk.num_rows new_times_ns = capture_times_ns[indices] pts_to_time.update(zip(old_pts.tolist(), new_times_ns.tolist())) else: # The sparse keyframe marker, emitted after every sample chunk. new_times_ns = np.array([pts_to_time[pts] for pts in old_pts.tolist()], dtype=np.int64) times = pa.array(new_times_ns.astype("datetime64[ns]")) # `metadata=` is load-bearing — see gotcha 3. new_field = pa.field(old_field.name, times.type, nullable=old_field.nullable, metadata=old_field.metadata) return Chunk.from_record_batch(batch.set_column(col_index, new_field, times))[0] return stream.map(_retag) ``` The cursor makes this order-dependent, so keep the retag on the single reader's stream, before any merge. This is the DROID loader's pattern — see the references. Any `map`/`flat_map` that assumes "every non-static chunk is samples, in order" has this same bug: it advances over the marker chunk and stamps it with whatever time comes next, silently moving the keyframe markers off their samples. ## Several cameras into one recording One reader per file, distinct entity paths, then merge: ```python streams = [ Mp4Reader(path, entity_path=f"/camera/{name}", timeline_name="real_time", timeline_type="timestamp").stream() for name, path in cameras.items() ] ( LazyChunkStream .merge(*streams) .collect(optimize=OptimizationProfile.OBJECT_STORE) .write_rrd(out_path, application_id=app_id, recording_id=segment_id) ) ``` `OBJECT_STORE` GOP-rebatches the video and preserves the reader's keyframe marker as-is; no `fix_keyframe=True` is needed. If you *do* see `skipping GoP rebatching … is_keyframe data is incorrect`, something upstream rewrote the marker (see gotcha 4) — `fix_keyframe=True` re-derives it from the encoded samples as an escape hatch. ## How the reader's chunks relate to optimize's The reader and `collect(optimize=…)` agree on **where GOPs start** but not on **how many GOPs share a chunk**, and that is by design: - The reader emits the **finest GOP-aligned partition**: exactly one chunk per GOP. It cannot do better, because it does not know which profile the data is headed for. - Optimize then applies the size policy, **merging consecutive GOPs** up to the profile's `max_bytes` / `max_rows`. It never splits a GOP across chunks, so every boundary it keeps is one the reader already produced. So optimize's partition is a pure *coarsening* of the reader's: same samples in the same order, every optimized chunk a run of whole reader GOPs, keyframe marker untouched. Where each GOP already sits near the profile's budget the two come out identical (a 30-frame clip at `gop_size=10` is 3 chunks either way). Where GOPs are small they diverge sharply — a 12-GOP clip becomes **1** chunk under `OBJECT_STORE` and **4** under `LIVE`. The practical consequence: **do not pre-merge or re-chunk the reader's output** to "help" optimize. Hand it the per-GOP stream and let the profile decide. And if you skip optimize entirely (writing straight through `send_chunks`, as the DROID loader does), you are storing the finest partition — correct, but more chunks than object storage wants. ### The optimize pass is cheap on GOP-chunked input Running optimize over the reader's output is not redundant work worth avoiding. On an 89 MB / 1800-frame / 60-GOP H.264 file: | | `collect()` | `collect(optimize=OBJECT_STORE)` | delta | | ----------------------------- | ----------- | -------------------------------- | ------------------- | | `chunk_by_gop=True` (default) | 11.1 ms | 16.3 ms | **+5.2 ms** (1.5×) | | `chunk_by_gop=False` | 14.2 ms | 48.6 ms | **+34.4 ms** (3.4×) | Neither half of the pass is expensive on GOP-aligned input: - **Detecting GOP starts is header-only.** `build_sample_index` runs `detect_gop_start` on every sample, but that reads a few bytes of each sample's header — it never decodes. The whole optimize delta above (5.2 ms) is smaller than a Python loop calling the same detector over the same 1800 samples (6.0 ms), which is mostly FFI overhead. - **Rebuilding the chunks copies nothing.** `chunk_from_gop` calls `taken(0..n)`, and `re_arrow_util::take_array` returns the array **uncopied** when the indices are consecutive from zero over the whole array. One chunk per GOP means that is always the case, so the multi-MB sample buffers are never duplicated — hence 89 MB "rebuilt" in 5 ms. That fast path is exactly what `chunk_by_gop=False` gives up: each GOP then spans 30 source chunks, so `chunk_from_gop` has to `concat_and_sort` them and the sample bytes really are copied. `LIVE` also costs more than `OBJECT_STORE` (+13.2 ms vs +5.2 ms on the same file), because its much smaller chunk budget makes it attempt compaction it cannot complete — a GOP is never split. Retag each camera *before* the merge — the retag above walks chunks with a cursor, so it must see one video's chunks in order. ## Gotchas 1. **Errors surface on the first pull, not at construction.** The constructor only checks that the file exists and validates its arguments; `stream()` builds a lazy pipeline and also succeeds. Codec support, keyframe layout, and FFmpeg availability are checked when the stream runs, so wrap the *consumption* (`to_chunks()`, `send_chunks`, iteration), not the `Mp4Reader(…)` call. 2. **A codec outside the five `VideoCodec` values cannot be a `VideoStream`.** `mp4v` (MPEG-4 Part 2) is the one you will actually meet in robot datasets; it raises `RuntimeError: MP4 error: MP4 demux: Video track uses unsupported codec "mp4v"`. Asset mode does accept the file, but only partly: the blob chunk is emitted and the `VideoFrameReference` index chunk is **skipped with a warning**, because the frame timestamps cannot be read either — so you get the bytes and no timeline. Skipping that camera and recording the fact as a recording property (what the DROID loader does) is usually better than a timeline-less blob, and either beats failing a whole episode over one camera. 3. **A `map`/`flat_map` that rewrites the time column must carry the field metadata over.** `pa.field(...)` without `metadata=old_field.metadata` drops `rerun:kind: 'index'`, and the rebuilt chunk silently becomes **static** — no error, no timeline, and the samples land outside time entirely. 4. **The trailing marker chunk is temporal but holds no samples**, so any per-chunk logic keyed on `not chunk.is_static` will process it as if it were samples. Key on the `VideoStream:sample` column instead. See the retag above. 5. **Asset mode is capped at ~2 GiB** by Arrow's i32 offsets, and duplicates the file's bytes into the RRD. 6. `mode="asset"` rejects both `chunk_by_gop=False` and `transcode=` with `ValueError` — those are stream-mode-only knobs. 7. `timeline_type="timestamp"` on its own shifts nothing. Without a retag, the video sits at the epoch. 8. **The reader emits compressed samples, never pixels.** Thumbnails, CLIP embeddings, or anything else needing RGB have to decode the file separately (OpenCV, PyAV); the `VideoStream` chunks cannot supply them. 9. Every `stream()` call re-decodes the file from scratch, and every terminal