Use when the user wants to find, download, or pick a NVIDIA Physical AI dataset on Hugging Face for autonomous-vehicle, robotics, spatial intelligence, manipulation, or neural-reconstruction workflows. Catalog of every dataset under huggingface.co/nvidia with the `PhysicalAI-` prefix, organised by domain (AV, Robotics-Manipulation, Robotics-GR00T, Robotics-mindmap, Robotics-NuRec, Spatial Intelligence, Grasping, Healthcare, Sim-Ready, Material properties), with per-dataset size, format, gating, license, and the downstream sibling skill (`ncore`, `nre`, `asset-harvester`, `nurec-fixer`) or upstream tool (Isaac Sim, CARLA, Isaac-GR00T, Cosmos-*) that consumes it. Do NOT use as a runtime — it routes you elsewhere. Trigger keywords: nvidia physical ai dataset, PhysicalAI- dataset, hf nvidia dataset, NCore dataset, NuRec dataset, GR00T dataset, GraspGen, SimReady, Cosmos-Drive-Dreams, Lyra SDG, Open-H-Embodiment, huggingface-cli download, physical_ai_av, dataset gated, RDS-HQ.
Use when the user wants to find, download, or pick a NVIDIA Physical AI dataset on Hugging Face for autonomous-vehicle, robotics, spatial intelligence, manipulation, or neural-reconstruction workflows. Catalog of every dataset under huggingface.co/nvidia with the `PhysicalAI-` prefix, organised by domain (AV, Robotics-Manipulation, Robotics-GR00T, Robotics-mindmap, Robotics-NuRec, Spatial Intelligence, Grasping, Healthcare, Sim-Ready, Material properties), with per-dataset size, format, gating, license, and the downstream sibling skill (`ncore`, `nre`, `asset-harvester`, `nurec-fixer`) or upstream tool (Isaac Sim, CARLA, Isaac-GR00T, Cosmos-*) that consumes it. Do NOT use as a runtime — it routes you elsewhere. Trigger keywords: nvidia physical ai dataset, PhysicalAI- dataset, hf nvidia dataset, NCore dataset, NuRec dataset, GR00T dataset, GraspGen, SimReady, Cosmos-Drive-Dreams, Lyra SDG, Open-H-Embodiment, huggingface-cli download, physical_ai_av, dataset gated, RDS-HQ.
version
0.1.0
tools
["Shell","Read","Write"]
license
CC-BY-4.0 AND Apache-2.0
dependencies
["bash","python","huggingface_hub"]
compatibility
All listed datasets are hosted on Hugging Face and require `huggingface_hub` (CLI: `huggingface-cli` / new `hf` shim) plus a HF user access token. Several datasets are GATED — they need explicit license acceptance on the HF dataset page before any token can pull them. Storage ranges from <1 GB to >100 TB (PhysicalAI-Autonomous-Vehicles, 133 TB).
Help the agent find, evaluate, and download an NVIDIA PhysicalAI-*
dataset on Hugging Face that fits the user's downstream task —
autonomous-vehicle reconstruction, robotics manipulation, GR00T
post-training, spatial-intelligence research, grasping, or sim-ready
content — and then hand off to the sibling skill (ncore, nre,
asset-harvester, nurec-fixer) or upstream NVIDIA tool that
actually consumes it.
Use this skill when: the user asks "is there an NVIDIA dataset
for X?", "where do I get NCore / NuRec / GR00T sample data?", or is
shopping the Hugging Face NVIDIA org for PhysicalAI-* collections.
Do NOT use this skill when:
The user already knows the dataset and just wants to run a
pipeline — jump straight to the consuming skill.
The user needs a non-NVIDIA dataset (Waymo, nuScenes, KITTI, …) —
this catalog is NVIDIA-only.
The user wants to train Cosmos / GR00T / Isaac Sim itself — that's
the upstream tool's job, not this catalog's.
Overview
Catalog of NVIDIA's open Physical AI dataset family on Hugging Face.
Pick by task (Section 2 § lookup table) or family (Sections 3–10).
Every entry lists: dataset path, size, format, license, gating, and the
downstream skill in this repo that consumes it.
Storage room sized to the dataset you're pulling (see the per-row
size column; some are < 1 GB, the AV dataset is 133 TB — always
pre-filter with --include or physical_ai_av).
Verifying secrets safely
Always check token presence with or a length-only
shell test; never write ad-hoc bash that interpolates
values. The common one-liner
hf auth whoami
HF_TOKEN
# BAD — leaks the secret to the terminal when the variable is setecho"HF_TOKEN: ${HF_TOKEN:+yes}${HF_TOKEN:-no}"
prints yes<token-value> whenever HF_TOKEN is set, because
${VAR:-no} only falls back to "no" when VAR is empty — when set
it expands to $VAR. Use one of these instead:
hf auth whoami# confirms the token without echoing ittest -n "$HF_TOKEN" && echo"HF_TOKEN: set (${#HF_TOKEN} chars)" || echo"HF_TOKEN: missing"
Have a HF user account that's logged in to the dataset page in a browser AT LEAST ONCE.
Have accepted any license / terms-of-use checkbox the dataset shows
(re-accept if the dataset has been re-gated — common for AV).
Three download patterns:
# Whole dataset (small / medium)
hf download nvidia/<dataset> --repo-type dataset --local-dir ./<dataset>
# Sub-folder only (recommended for large multi-task collections)
hf download nvidia/PhysicalAI-Robotics-GR00T-X-Embodiment-Sim \
--repo-type dataset \
--include "gr1_arms_only.CanSort/**" \
--local-dir ./gr00t_dataset
# Sparse-checkout via git-LFS (if you want incremental git-style work)
git clone --filter=blob:none --no-checkout https://huggingface.co/datasets/nvidia/<dataset>
cd <dataset>
git sparse-checkout init --cone
git sparse-checkout set <subfolder>
git checkout main
Special-case downloaders:
PhysicalAI-Autonomous-Vehicles (133 TB) — use the official Python
toolkit pip install physical_ai_av
(NVlabs/physical_ai_av)
to filter by sensor / country / split before downloading; otherwise
you will pull TBs you don't need.
Default policy: pre-filter to platform_class == 'hyperion_8.1'
before any bulk pull. The dataset mixes two sensor rigs
(hyperion_8 and hyperion_8.1) and the entire downstream
NuRec / NCore / Asset-Harvester chain in this repo is only
validated for hyperion_8.1. Roughly half of the 306k clips fall
outside that platform, so the filter typically cuts the working set
(and download size) in half.
Apply this filter by default in any download script you generate.
Only skip the filter when the user explicitly asks for it (e.g.
"download all platforms", "include hyperion_8", "don't filter by
platform") or when they hand you a specific clip UUID — in that
single-clip case, download_clip_features(clip_id=...) is already
scoped and no platform filter is needed.
from physical_ai_av import PhysicalAIAVDatasetInterface
dataset = PhysicalAIAVDatasetInterface()
dataset.download_metadata()
dc = dataset.metadata['data_collection']
hyperion_81_clip_ids = dc[dc['platform_class'] == 'hyperion_8.1'].index.tolist()
PhysicalAI-Autonomous-Vehicle-Cosmos-Drive-Dreams (3 TB) — use the
upstream download.py
(nv-tlabs/Cosmos-Drive-Dreams)
with --file_types {hdmap,lidar,synthetic} to select layers.
PhysicalAI-SpatialIntelligence-Lyra-SDG (25 TB) — hf download --local-dir lyra_dataset/tar; untar each tar yourself.
Spatial-Intelligence-Warehouse — chunked TAR-GZs need a manual
loop after download (script provided in the upstream card).
For dataset filtering / preview: NVIDIA's
Cosmos Dataset Search (CDS)
lets you query a 41K subset of the AV dataset semantically before
downloading.
Filtered AV download recipe
Always use this recipe (or a derivative of it) when the user asks to
download raw clips from PhysicalAI-Autonomous-Vehicles. The
hyperion_8.1 filter is the default; it matches every downstream
skill in this repo (ncore, nre, asset-harvester, nurec-fixer).
When to skip the platform filter:
The user explicitly opts out — e.g. "download all platforms",
"include hyperion_8 too", "ignore platform_class", or asks for a
dataset-wide statistic. In that case, drop the platform mask and
warn them that NuRec / NCore tooling will not work on the
hyperion_8 clips.
The user gave you a specific clip UUID. A single-clip
download_clip_features(clip_id=...) is already scoped, and
filtering by platform_class for one clip is pointless.
Recipe (defaults: filter on; user may layer extra masks like country
or sensor presence on top):
from physical_ai_av import PhysicalAIAVDatasetInterface
dataset = PhysicalAIAVDatasetInterface()
dataset.download_metadata()
dc = dataset.metadata['data_collection']
clip_mask = dc['platform_class'] == 'hyperion_8.1'# Optional extra masks (only add when the user asked for them):# clip_mask &= dc['country'] == 'US'# sp = dataset.metadata['feature_presence'] # 26.03+; was 'sensor_presence' in 25.10# clip_mask &= sp['lidar_top_360fov']
clip_ids = dc[clip_mask].index.tolist()
print(f"Downloading {len(clip_ids)} hyperion_8.1 clips")
dataset.download_clip_features(
clip_id=clip_ids,
features=["camera_front_wide_120fov", "lidar_top_360fov", "egomotion"],
max_workers=8,
)
Single-clip fast path (no platform filter — the UUID is already
specific):
Per-sensor parquet/mp4 chunks of ~100 clips; UUIDs cross-link sensors
License
NVIDIA AV Dataset License Agreement (gated; AV-development-only purpose; no biometric / surveillance / re-identification; expires 12 months after download)
Toolkit
pip install physical_ai_av — direct filtered downloads + format docs. Default to filtering metadata['data_collection']['platform_class'] == 'hyperion_8.1' before any bulk pull (see Filtered AV download recipe); only skip the filter on explicit user request or when downloading a specific clip UUID.
Versions
26.03 (current; offline-optimized features for 97 % of clips), 25.10 (initial)
../ncore/SKILL.md (convert raw clips to NCore V4), ../asset-harvester/SKILL.md (extract per-object Gaussian assets). NuRec workflows are only validated for platform_class == hyperion_8.1. Upstream sim/training tools without an in-repo skill: NVlabs/alpamayo-1.5, NVlabs/alpasim, CARLA.
../nre/SKILL.md (render the USDZs locally or over serve-grpc), ../nurec-fixer/SKILL.md (clean up rendered frames). Upstream consumer without an in-repo skill: CARLA (NuRec integration in 0.9.16+).
PhysicalAI-Autonomous-Vehicle-Cosmos-Drive-Dreams
Cosmos-Transfer-style synthetic + HD-map labels for diverse weather.
Pointer / placeholder — content moved to PhysicalAI-Autonomous-Vehicle-Cosmos-Drive-Dreams. Use that. (Card is 2.59 kB.)
Robotics — Manipulation
All in LeRobot v2.x format unless noted, generated in IsaacSim with
task-and-motion planning + scene_synthesizer procedural scenes +
CuRobo motion generation.
PhysicalAI-Robotics-Manipulation-Kitchen
Bimanual Kinova Gen3 in procedurally-generated kitchens.
Upstream consumers without an in-repo skill: nvidia/Cosmos-Transfer1 (legacy Transfer1 workflow), Isaac Sim / Isaac Lab (replay scripts ship in the dataset repo).
PhysicalAI-Robotics-Manipulation-Kitchen-Demos
Massive human-teleop dataset on Franka + Omron mobile base.
Spatial-memory benchmark from nvidia-isaac/nvblox_mindmap. Each
dataset is one task with the same multimodal layout (RGB-D + camera
intrinsics/poses + nvblox vertex features in .zst + robot state).
All four are CC-BY-NC-4.0 (research only). Models trained:
nvidia/PhysicalAI-Robotics-mindmap-Checkpoints.
../nre/SKILL.md to retrain reconstructions; upstream Isaac Sim 5.1 (no in-repo skill) for AMR simulation; pair with MobilityGen for AMR data generation.
PhysicalAI-NuRec-PPISP
Photometric-variation benchmark for radiance-field methods.
ls <local-dir>
du -sh <local-dir> # confirm size matches the table above (within 10 %)# For LeRobot v2.x:
python -c "from lerobot.common.datasets.lerobot_dataset import LeRobotDataset; \
d = LeRobotDataset('<local-dir>'); print(d.meta.info)"# For NCore V4 (.zarr.itar):
ncore_vis <local-dir>/clips/<uuid>/pai_<uuid>.json # via ../ncore/SKILL.md# For USDZ scenes:# load in Isaac Sim 5.1 — File → Open → <scene>.usdz
GREEN when:
File counts and total size match the dataset card to within ~10 %.
For gated datasets, the download did not silently terminate at the
license-agreement page (re-hf auth login if the first chunk is HTML).
For LeRobot datasets, meta/info.json parses and the episode count
matches the card.
Troubleshooting
Repo gated. Cannot access … 401 — open the dataset URL in a
browser, click Agree on the license / contact-info form, then retry
with the same token. Tokens don't get auto-refreshed when a new
agreement appears (re-accept after major version bumps).
First chunk is 5 KB of HTML — same cause as above.
Download hangs indefinitely on AV (133 TB) — you almost certainly
don't want the whole thing. Use physical_ai_av to filter
platform_class == 'hyperion_8.1' (default — see
Filtered AV download recipe), plus
any sensor / country / split mask, BEFORE pulling.
AV clip downloaded but downstream NCore / NuRec / Asset-Harvester
rejects it — double-check data_collection['platform_class'] for
that clip; only hyperion_8.1 is validated. If it's hyperion_8,
either swap to a hyperion_8.1 clip or accept that the downstream
reconstruction tooling will not work.
Cosmos-Drive-Dreams 3 TB on a small disk — pass
--file_types synthetic (700 GB) or --file_types hdmap (small) to
the official download.py.
Lyra-SDG 25 TB out of disk — hf download --include "tar/static_*" --exclude "tar/dynamic_*" to take just the 3D half (or vice-versa).
Cosmos-Synthetic looks empty (2.59 kB) — it's a pointer page
redirecting to Cosmos-Drive-Dreams. Use that.
mindmap dataset only has 10 demos when the card says 200 / 1000 —
expected; regenerate the rest from the HDF5 via the upstream
mindmap data-generation docs.
AV license expired (12 months) — re-accept on the HF page; this
is by-design per the License Agreement section 7.
Limitations
Catalogue-only. This skill does not actually run any
pipeline — it routes the agent to a sibling skill once the right
dataset is chosen.
NVIDIA-only. Non-NVIDIA datasets (Waymo, nuScenes, KITTI,
PandaSet, …) are out of scope; the ncore skill covers how to
convert those into NCore V4 manually.
Gating is enforced by Hugging Face, not by this skill. Several
datasets (e.g. PhysicalAI-Autonomous-Vehicles*,
nvidia/Fixer-adjacent assets, mindmap large variants) require
both license acceptance on the HF page and a valid HF_TOKEN.
Acceptances expire (AV is annual) and must be re-clicked.
Footprint can be enormous. PhysicalAI-Autonomous-Vehicles is
~133 TB; Lyra-SDG ~25 TB; Cosmos-Drive-Dreams ~3 TB. Always check
free disk and use --include / --exclude filters before kicking
off a full pull.
The catalogue drifts. NVIDIA ships new PhysicalAI-* datasets
regularly. If a dataset isn't in this skill, browse
https://huggingface.co/nvidia?search_models=PhysicalAI- and
consider opening a PR to update Section 2 / the relevant family.
No GPU / runtime checks here. Storage and HF auth are the only
prerequisites this skill validates; per-dataset compute needs
belong to the consuming sibling skill.
