name	add-exoego-dataset
description	Add or update SimpleCV ExoEgo dataset adapters end to end. Use when implementing a dataset loader, mapping labels to COCO-133, storing MANO/depth metadata, saving RRDs, registering ExoEgo Forge catalog entries, serving the Rerun catalog/web viewer, or validating screenshots.

Add ExoEgo Dataset

Keep this skill short: read the repo, copy local patterns, prove the result visually, and avoid inventing parallel tooling.

Read First

Task/goal doc, if present.
AGENTS.md, pyproject.toml, and docs/exoego_schema.md.
Closest existing adapter: usually HOCap for ego+exo+MANO, Assembly101 for multi-view hand keypoints, HOT3D for headset+MANO, or Aria/EgoDex for ego-only.
Existing entrypoints before adding new ones: view_exoego.py, batch_raw_to_rrd.py, and exoego_forge_catalog.py.

Workflow

Use Pixi and TDD. Build vertical slices: identity/config, discovery count, load_labels=False, labels, optional MANO/depth/mesh, single RRD, catalog, full conversion, screenshots.
Inspect real files early. Trust headers and sample arrays over docs for names, capitalization, timestamps, units, and shapes.
Prefer the established three-file split for ego+exo datasets: data/exoego/<dataset>.py, data/ego/<dataset>_ego.py, data/exo/<dataset>_exo.py.
Add SequenceIdentity, config _target, stream timestamps, load_labels, dataset iteration/counting, dataset_defaults, and catalog defaults when full registration is in scope.
Do not add dataset-specific viewer/catalog scripts. Use generic view, batch conversion, and tools/catalog.py.

Labels And MANO

End state for labels is ExoEgoLabels.xyzc_stack: Float[np.ndarray, "num_frames 133 4"]; missing points are NaN xyz with 0.0 confidence.
COCO-133 rows: body 0:17, feet 17:23, face 23:91, left hand 91:112, right hand 112:133.
Use released 3D keypoints when available. Store MANO/SMPL parameters separately; do not generate COCO labels from MANO/SMPL unless this repo already has a tested conversion for that exact source.
ManoStack convention is hand 0=right, 1=left. If a dataset has one shared shape requirement but per-hand shapes differ, follow the user/local precedent and warn rather than fail.
Verify hand order from upstream code or links before reordering. Apply MANO-to-COCO reorder only to raw MANO joints, not released COCO-ordered hand keypoints.

Rerun And Catalog Lessons

Log video through the shared log_video path so recordings use video streams, not ad hoc asset-video logging.
Tune image_plane_distance from screenshots; overlapping camera frusta usually mean it is too large.
For browser sharing, use one SimpleCV catalog command but expose two endpoints: web viewer port and catalog server port. Tailscale raw TCP forwarding worked where HTTPS proxying caused malformed catalog responses.
Rerun directory dataset registration expects direct child .rrd files. For nested datasets, register a flat hardlink mirror so built-in dataset tables show rows.
If AV1/Hugging Face mirroring is in scope, transcode videos first, prefer NVENC when available, register against AV1 paths, and skip sidecars made obsolete by the mirror.

Done Means

Focused tests pass, including catalog tests for defaults, camera names, table names, and nested RRD discovery.
A representative sequence opens with no labels, then with labels, then as an RRD.
Full conversion covers every supported session; every skip has a written reason.
Native Rerun screenshots show direct viewer, labels/MANO when present, catalog table rows, and clicked recordings.

Add ExoEgo Dataset

Keep this skill short: read the repo, copy local patterns, prove the result visually, and avoid inventing parallel tooling.

Read First

Task/goal doc, if present.

AGENTS.md, pyproject.toml, and docs/exoego_schema.md.

Closest existing adapter: usually HOCap for ego+exo+MANO, Assembly101 for multi-view hand keypoints, HOT3D for headset+MANO, or Aria/EgoDex for ego-only.

Existing entrypoints before adding new ones: view_exoego.py, batch_raw_to_rrd.py, and exoego_forge_catalog.py.

Workflow

Use Pixi and TDD. Build vertical slices: identity/config, discovery count, load_labels=False, labels, optional MANO/depth/mesh, single RRD, catalog, full conversion, screenshots.

Inspect real files early. Trust headers and sample arrays over docs for names, capitalization, timestamps, units, and shapes.

Prefer the established three-file split for ego+exo datasets: data/exoego/<dataset>.py, data/ego/<dataset>_ego.py, data/exo/<dataset>_exo.py.

Add SequenceIdentity, config _target, stream timestamps, load_labels, dataset iteration/counting, dataset_defaults, and catalog defaults when full registration is in scope.

Do not add dataset-specific viewer/catalog scripts. Use generic view, batch conversion, and tools/catalog.py.

Labels And MANO

End state for labels is ExoEgoLabels.xyzc_stack: Float[np.ndarray, "num_frames 133 4"]; missing points are NaN xyz with 0.0 confidence.

COCO-133 rows: body 0:17, feet 17:23, face 23:91, left hand 91:112, right hand 112:133.

Use released 3D keypoints when available. Store MANO/SMPL parameters separately; do not generate COCO labels from MANO/SMPL unless this repo already has a tested conversion for that exact source.

ManoStack convention is hand 0=right, 1=left. If a dataset has one shared shape requirement but per-hand shapes differ, follow the user/local precedent and warn rather than fail.

Verify hand order from upstream code or links before reordering. Apply MANO-to-COCO reorder only to raw MANO joints, not released COCO-ordered hand keypoints.

Rerun And Catalog Lessons

Log video through the shared log_video path so recordings use video streams, not ad hoc asset-video logging.

Tune image_plane_distance from screenshots; overlapping camera frusta usually mean it is too large.

For browser sharing, use one SimpleCV catalog command but expose two endpoints: web viewer port and catalog server port. Tailscale raw TCP forwarding worked where HTTPS proxying caused malformed catalog responses.

Rerun directory dataset registration expects direct child .rrd files. For nested datasets, register a flat hardlink mirror so built-in dataset tables show rows.

If AV1/Hugging Face mirroring is in scope, transcode videos first, prefer NVENC when available, register against AV1 paths, and skip sidecars made obsolete by the mirror.

Done Means

Focused tests pass, including catalog tests for defaults, camera names, table names, and nested RRD discovery.

A representative sequence opens with no labels, then with labels, then as an RRD.

Full conversion covers every supported session; every skip has a written reason.

Native Rerun screenshots show direct viewer, labels/MANO when present, catalog table rows, and clicked recordings.

add-exoego-dataset