launcher-config

name	launcher-config
description	Configure NeMo AutoModel job launches for interactive runs, Slurm clusters, and SkyPilot cloud execution.
when_to_use	Configuring Slurm or SkyPilot job submission, setting up multi-node launch scripts, debugging job submission failures, or switching between interactive and cluster launch modes.

Launcher Configuration

NeMo AutoModel supports three launch methods: interactive (torchrun), Slurm (HPC clusters), and SkyPilot (cloud-agnostic).

Launch Methods

1. Interactive (default): runs torchrun on the current node. Suitable for single-node development and debugging.
2. Slurm: submits a batch job to an HPC cluster scheduler. Handles multi-node setup, container management, and environment configuration.
3. SkyPilot: cloud-agnostic job submission to AWS, GCP, Azure, Lambda, or Kubernetes. Supports spot instances.

Interactive Launch

# Single GPU
automodel finetune llm -c config.yaml

# Multi-GPU (all GPUs on current node)
torchrun --nproc_per_node=8 -m nemo_automodel._cli.app finetune llm -c config.yaml

No additional YAML section is needed for interactive mode. The CLI routes to torchrun automatically when no slurm: or skypilot: section is present in the config.

Slurm Configuration

The SlurmConfig dataclass generates an SBATCH script from a template.

YAML Example

slurm:
  job_name: llm_finetune
  nodes: 2
  ntasks_per_node: 8
  time: "04:00:00"
  account: my_account
  partition: batch
  container_image: nvcr.io/nvidia/nemo:dev
  hf_home: ~/.cache/huggingface
  extra_mounts:
    - source: /data
      dest: /data
  env_vars:
    WANDB_API_KEY: "${WANDB_API_KEY}"
    HF_TOKEN: "${HF_TOKEN}"

Key Fields

• job_name: Slurm job identifier
• nodes: number of nodes to request
• ntasks_per_node: number of tasks (GPUs) per node
• time: wall-time limit in HH:MM:SS format
• account, partition: Slurm scheduling parameters
• container_image: Enroot/Pyxis container image path
• nemo_mount: mount point for NeMo AutoModel source inside the container
• hf_home: HuggingFace cache directory path
• extra_mounts: list of VolumeMapping(source, dest) for additional container bind mounts
• master_port: port for distributed communication (default 13742)
• env_vars: environment variables passed into the job
• nsys_enabled: when true, wraps the training command with nsys profile for Nsight Systems profiling

SkyPilot Configuration

The SkyPilotConfig dataclass defines cloud job parameters.

YAML Example

skypilot:
  cloud: aws
  accelerators: "H100:8"
  num_nodes: 2
  use_spot: true
  disk_size: 200
  region: us-east-1
  setup: "pip install nemo-automodel"
  env_vars:
    HF_TOKEN: "${HF_TOKEN}"

Key Fields

• cloud: target cloud provider (aws, gcp, azure, lambda, kubernetes)
• accelerators: GPU type and count (e.g., "H100:8", "A100-80GB:4")
• num_nodes: number of cloud instances
• use_spot: use preemptible/spot instances for cost savings
• disk_size: disk size in GB per node
• region: cloud region for instance placement
• setup: shell commands to run before the training job (e.g., install dependencies)
• env_vars: environment variables for the job

Multi-Node Environment

For multi-node training (both Slurm and SkyPilot), the launcher automatically configures:

• MASTER_ADDR: hostname of the first node
• MASTER_PORT: port for rendezvous (default 13742)
• WORLD_SIZE: total number of processes (nodes * ntasks_per_node)
• NCCL environment variables for optimized collective communication

Nsys Profiling

Enable Nsight Systems profiling in Slurm jobs:

slurm:
  nsys_enabled: true

This wraps the training command with nsys profile, producing a .nsys-rep file for performance analysis.

Code Anchors

• components/launcher/slurm/config.py - SlurmConfig dataclass, VolumeMapping
• components/launcher/slurm/template.py - SBATCH script template generation
• components/launcher/slurm/utils.py - Slurm submission utilities
• components/launcher/skypilot/config.py - SkyPilotConfig dataclass
• _cli/app.py - CLI entry point and launcher routing logic

Pitfalls

• Port collisions: if the default master_port (13742) is in use by another job on the same node, change it to avoid connection failures.
• Container mounts: the source path in extra_mounts must exist on all nodes in the allocation. Missing paths cause container startup failures.
• Slurm fault tolerance: the fault tolerance plugin is Slurm-specific and does not work with SkyPilot or interactive mode.
• SkyPilot spot preemption: spot instances (use_spot: true) may be preempted by the cloud provider. Enable checkpointing with short intervals to minimize lost work.
• Environment variable syntax: use ${VAR} syntax in YAML for shell variable expansion. Bare variable names will not be expanded.
• Time limit vs async checkpoint: if the Slurm time limit is too short, an in-progress async checkpoint write may be killed before completion, resulting in a corrupted checkpoint. Leave at least 5-10 minutes of margin.