| name | habitat-gs-train |
| description | Train and evaluate a navigation policy in the habitat-gs simulator. Covers the full generate-episodes → train → evaluate flow for PointNav / ImageNav / ObjectNav (Habitat-Lab + DDPPO reinforcement learning) and for Vision-and-Language Navigation (StreamVLN, Uni-NaVid). Use when the user wants to train, fine-tune, resume, or evaluate a nav policy / agent on GS scenes — NOT for interactively piloting a live sim (use the habitat-gs-control skill for that). |
habitat-gs-train
Train and evaluate navigation policies on photo-realistic 3D Gaussian Splatting scenes in habitat-gs. Five tasks are supported, each with a one-click generate → train → evaluate pipeline driven by scripts under scripts_gs/.
IMPORTANT — run from the repo root. Every command below assumes the current directory is the habitat-gs/ project root and that the right conda env is active. The scripts cd to the project root themselves, but paths like data/scene_datasets/gs_scenes/... are repo-relative. Never invent flags — the exact flags each script accepts are in references/.
Pick the task first
| Task | Kind | Conda env | Backbone | Reference |
|---|
| PointNav | RL (Habitat-Lab + DDPPO) | habitat-gs | PointNavResNet-50 + LSTM | references/task-pointnav.md |
| ImageNav | RL (Habitat-Lab + DDPPO) | habitat-gs | PointNavResNet-50 + LSTM | references/task-imagenav.md |
| ObjectNav | RL (Habitat-Lab + DDPPO) | habitat-gs | PointNavResNet-50 + LSTM | references/task-objectnav.md |
| StreamVLN | VLN (VLM supervised fine-tune) | habitat-gs-streamvln | LLaVA-Video-7B-Qwen2 + SigLIP | references/task-streamvln.md |
| Uni-NaVid | VLN (VLM supervised fine-tune) | habitat-gs-uni-navid | Vicuna-7B + EVA-ViT-G | references/task-uninavid.md |
RL tasks (PointNav / ImageNav / ObjectNav) are the lightweight, fast path — they run in the base habitat-gs env and need no external repo. Start here for "quickly train and evaluate a navigation policy."
VLN tasks (StreamVLN / Uni-NaVid) are heavy: each needs ≥80 GB VRAM/GPU (StreamVLN has a 24 GB LoRA mode), a separate cloned conda env, and an external repo cloned as a sibling of habitat-gs/. Only go here when the user explicitly wants instruction-following VLN.
The universal flow (all tasks)
- Prerequisites — env + Habitat-Lab + GS data. Read
references/prerequisites.md and verify before anything else. Skipping this is the #1 cause of failures.
- Generate episodes / trajectories — produces the dataset the policy trains/evals on. The released dataset already ships episodes, so this is OPTIONAL for the standard scenes (the train/eval scripts abort with a clear message if the data is missing).
- Train —
bash scripts_gs/train_<task>.sh --output <dir> [options].
- Evaluate —
bash scripts_gs/eval_<task>.sh --ckpt <ckpt> [options]; read where metrics land in references/outputs-and-metrics.md.
Quick start (PointNav, the simplest task)
conda activate habitat-gs
python scripts_gs/generate_pointnav_episodes.py
bash scripts_gs/train_pointnav.sh --output output/pointnav
bash scripts_gs/eval_pointnav.sh --ckpt output/pointnav/checkpoints/ckpt.0.pth
ImageNav and ObjectNav are identical in shape — swap pointnav for imagenav / objectnav (their default step counts and episode generators differ; see their references).
How to use this skill
- Confirm the task with the table above. If the user just says "train a nav policy", default to PointNav (fastest to a working result) and say so.
- Check prerequisites (
references/prerequisites.md): correct conda env, Habitat-Lab installed with the numpy pin patched, GS data under data/scene_datasets/gs_scenes/, and for ObjectNav generation the SAM + CLIP checkpoints.
- Decide whether to generate data. If
data/scene_datasets/gs_scenes/episodes/<task>/{train,val}/content already exists, skip generation. Otherwise run the generator (read the task reference — ObjectNav needs extra models; VLN needs a VLM endpoint).
- Launch training with
train_<task>.sh. Training is long-running — launch it in the background and poll, or hand the user the exact command. For multi-GPU pass --num-gpus N --num-envs M. See references/training-and-finetuning.md for resume vs. fine-tune vs. encoder-transfer.
- Evaluate with
eval_<task>.sh --ckpt <checkpoint.pth>. Prefer a single checkpoint file — the directory form is a polling watcher that can hang (see references/task-pointnav.md). Add --video-dir DIR (RL) / --save-video (VLN) to record rollouts. For a quick check, append habitat_baselines.test_episode_count=3.
- Report metrics by reading them from the eval output (RL: SPL / Success / DistanceToGoal printed to stdout &
eval.stdout; VLN: SR / SPL / OSR / DTG, Uni-NaVid also writes summary.json). See references/outputs-and-metrics.md.
Reference index
| File | Contents |
|---|
references/prerequisites.md | Conda envs, Habitat-Lab install + numpy patch, GS data download, ObjectNav SAM/CLIP models |
references/data-layout.md | The shared gs_scenes/ data root, episode .json.gz schema, where every dataset file lives |
references/task-pointnav.md | PointNav generate / train / eval, flags, defaults |
references/task-imagenav.md | ImageNav generate / train / eval, the goal-image quality gate |
references/task-objectnav.md | ObjectNav generate (SAM+CLIP, indoor/outdoor, InteriorGS GT variant) / train / eval |
references/task-streamvln.md | StreamVLN one-time setup, episode + trajectory gen, 3-stage train (+LoRA), eval |
references/task-uninavid.md | Uni-NaVid one-time setup, trajectory gen, stage-1/2 train, eval |
references/training-and-finetuning.md | DDPPO config knobs, resume, fine-tune, encoder transfer, multi-GPU, configs dir |
references/outputs-and-metrics.md | Output run-dir layout, checkpoints, TensorBoard, where eval metrics appear |
references/troubleshooting.md | Common errors and fixes; the cross-task smoke test |
Smoke test the whole pipeline
scripts_gs/_verify_full_pipeline.sh runs train+eval for all five tasks on a tiny 3-train / 3-val subset. It is the authoritative end-to-end command reference, but it hardcodes a foreign conda path and work dir — edit those before running. Details in references/troubleshooting.md.
Troubleshooting (quick)
| Problem | Solution |
|---|
Training episode data not found | Run the task's generate_* script, or download the released Nav Data into data/scene_datasets/gs_scenes/episodes/ |
ModuleNotFoundError: No module named 'habitat' | Habitat-Lab not installed in this env — see references/prerequisites.md |
AssertionError: ... number of environments (1) ... mini batches (2) | Training needs --num-envs >= 2; --num-envs 1 is eval-only. See references/training-and-finetuning.md |
numpy version conflict on pip install -e habitat-lab | Patch habitat-lab/habitat-lab/requirements.txt: numpy==1.26.4 → numpy>=2.0.0,<2.4 |
| GS scene renders blank / CUDA error | habitat-gs must be built with HABITAT_WITH_CUDA=ON; GS rendering requires CUDA |
| VLN env / external repo errors | Run the task's setup_*.sh first; StreamVLN/Uni-NaVid must be cloned as siblings of habitat-gs/ |
