asset-harvester

name	asset-harvester
description	Use to install and run NVIDIA Asset Harvester (Apache-2.0) to extract per-object 3D Gaussian Splat assets (`gaussians.ply`) from AV NCore V4 clips or masked single images via SparseViewDiT + TokenGS, optionally producing `metadata.yaml` for NuRec object insertion. Do NOT use for full-scene reconstruction (use `nre`) or for inputs without per-object masks.
version	0.1.1
tools	["Shell","Read","Write"]
license	CC-BY-4.0 AND Apache-2.0
compatibility	Linux + conda (Miniconda/Miniforge), NVIDIA driver >= 570 (CUDA 12.8), GCC 10-13, CUDA toolkit 12.8 (installed by setup.sh), ~16 GB GPU VRAM (use `--offload_model_to_cpu` for less). HF_TOKEN required for gated `nvidia/asset-harvester` checkpoints. Egress to github.com, huggingface.co, pypi.org, download.pytorch.org.
dependencies	["bash","conda","git","python3"]
metadata	{"author":"NVIDIA NRS <[e-mail]>","tags":["asset-harvester","autonomous-vehicles","3d-reconstruction","gaussian-splatting","simulation"],"upstream":"https://github.com/NVIDIA/asset-harvester","project_page":"https://research.nvidia.com/labs/sil/projects/asset-harvester/","paper":"https://arxiv.org/abs/2604.18468","hf_model":"https://huggingface.co/nvidia/asset-harvester","hf_demo":"https://huggingface.co/spaces/nvidia/asset-harvester","hf_dataset":"https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NCore","hf_benchmark":"https://huggingface.co/datasets/nvidia/NuRec-AV-Object-Benchmark","time-estimate":"45min (20min setup + 15min inference + evaluation)"}

Asset Harvester

Purpose

Install and drive NVIDIA Asset Harvester to extract per-object 3D Gaussian Splat assets from sparse autonomous-vehicle (AV) object observations — either a multi-view crop pulled from an NCore V4 driving log or a single masked image. The output is a simulation-ready gaussians.ply plus optional metadata.yaml that NVIDIA Omniverse NuRec can ingest as an external asset. Apache-2.0 upstream code lives at https://github.com/NVIDIA/asset-harvester.

When to Use / When NOT to Use

Use this skill when:

• The user has AV clips or masked single images and wants per-object 3D assets via the SparseViewDiT + TokenGS pipeline.
• The user has NCore V4 driving-log clips and wants per-track 3D assets for simulation.
• The user asks about SparseViewDiT, TokenGS, or wants to reproduce the Asset Harvester paper / HF Space demo locally.
• The user wants .ply Gaussians + metadata.yaml suitable for NVIDIA Omniverse NuRec object insertion.

Do NOT use this skill when:

• The user wants a full-scene reconstruction (use the nre skill).
• The user has no per-object masks or AV-style object crops.
• The user wants text-to-3D, indoor scans, or non-AV imagery — out of distribution.
• The user wants to ingest raw sensor data into NCore V4 (use the ncore skill first).
• The user wants to re-train SparseViewDiT or TokenGS — this skill is install + inference only.
• The user just wants the no-install demo: point them at https://huggingface.co/spaces/nvidia/asset-harvester.

Background

Open-source (Apache-2.0) image-to-3D pipeline pairing SparseViewDiT (multiview diffusion, 16 consistent views) with TokenGS (feed-forward Gaussian lifting):

NCore V4 clip ──► NCore parsing ──► SparseViewDiT (16-view diffusion)
              ──► TokenGS lifting ──► gaussians.ply
              ──► (optional) metadata.yaml for NuRec object insertion

Single HF repo nvidia/asset-harvester ships four checkpoints: AH_object_seg_jit.pt (AV-object Mask2Former), AH_multiview_diffusion.safetensors (SparseViewDiT), AH_camera_estimator.safetensors (camera pose, used when calibration is absent), and (TokenGS).