alpamayo1-5-sft

name	alpamayo1-5-sft
description	Run end-to-end supervised fine-tuning (SFT) of the Alpamayo-1.5 VLM action model on PAI (with navigation conditioning) or LingoQA (for visual question answering). The agent collects a small set of choices from the user up front (Stage-1 task — nav / vqa / default trajectory, W&B preference, dataset and checkpoint paths, which stage(s) to run), then drives the whole pipeline. Use when an agent must convert the released `nvidia/Alpamayo-1.5-10B` checkpoint into A1-format with `convert_checkpoint.py to-a1`, prepare PAI + nav annotations or LingoQA Scenery, run Stage-1 VLM SFT on the chosen task, optionally continue into Stage-2 trajectory-diffusion-expert training (for trajectory tasks), and evaluate; when setting up the `a1_5_sft` uv venv from scratch; when overriding Hydra config (`vla_processor` variant, `chunk_ids`, learning rate, DeepSpeed); when diagnosing common SFT failures (flash-attn dtype warnings, wandb 403, DeepSpeed grad-accum mismatch, `use_cache` checkpointing warnings, hardcoded-path slip-ups in the shipped config). Trigger keywords: alpamayo, alpamayo-1.5, alpamayo1_5, alpamayo1.5, sft, post-train, post-training, fine-tune, finetune, vlm, action expert, trajectory diffusion, reasoning vla, qwen3-vl, qwen3, pai, physical_ai_av, physicalai-autonomous-vehicles, lingoqa, lingo qa, scenery, vqa, visual question answering, navigation, nav, route, hydra, omegaconf, deepspeed, zero2, torchrun, accelerate, flash-attn, flash attention, transformers, huggingface, Alpamayo-1.5-10B, Alpamayo-R1-10B, convert_checkpoint, to-a1, A1-format, min_ade, ade, corner_distance, wandb, hydra override, hydra full error, use_cache, gradient checkpointing, ddp_find_unused_parameters, pretrained_model_name_or_path, stage1_vlm_checkpoint_path, uv sync, PAIDatasetWithNav, LingoQADataset, vla_processor, gdown.
license	Apache-2.0
metadata	{"author":"nvidia","version":"2026.05"}

Alpamayo-1.5 SFT (Post-training)

This skill teaches an agent to fine-tune Alpamayo-1.5 end-to-end. The recipe at recipes/alpamayo1_5_sft/ extends the Alpamayo-1 SFT pipeline with two new Stage-1 task flavours:

1. Navigation-conditioned trajectory — trajectory prediction guided by route instructions (e.g. "Turn left in 40m"), trained on PAI with a separate nav-annotation JSON.
2. Visual Question Answering — driving-scene Q&A, demonstrated on the LingoQA Scenery split.

A default-trajectory Stage-1 variant (same pattern as Alpamayo-1 SFT) is also available. Stage-2 trajectory-diffusion-expert training is supported on top of the trajectory-style Stage-1 outputs (nav or default). Both stages share one entry point (train_hf.py) and one base config (configs/sft_base.yaml). Evaluation runs through evaluate_hf.py. Validated on 8× H100 80 GB.

Table of Contents

1. When to use this skill
2. Inputs to collect from the user (ask once, up front)
3. Mental model — what gets trained, in what order
4. Install — a1_5_sft venv
5. Checkpoint — download Alpamayo-1.5 and convert to A1 format
6. Dataset — PAI (nav / trajectory) or LingoQA (VQA)
7. VLA processor variants
8. Stage 1 — Nav-conditioned trajectory
9. Stage 1 — LingoQA VQA
10. Stage 2 — Action expert (trajectory diffusion)
11. Evaluation
12. Logging — W&B, TensorBoard, offline
13. Multi-GPU, multi-node, and cluster runs
14. Hydra override cheatsheet