nemo-customizer

# NeMo Customizer End-to-end **SFT + LoRA** (automodel/unsloth), **DPO**, and **GRPO** (rl) on NeMo Helix. Three backend plugins ship in this repo — all submit through `nemo customization <plugin> --job-json ...` and expose no local `run` verb: | Backend | Invocation | Trains | Where it runs | Pick when | |---------|------------|--------|---------------|-----------| | **`automodel`** (default) | `automodel --job-json` | SFT / LoRA | **Kubernetes GPU pods or the platform's Docker GPU executor** — whichever the platform provides. Multi-node (`num_nodes > 1`) needs Kubernetes | General SFT/LoRA; multi-GPU (data/tensor parallel); distillation; full-weight SFT | | **`unsloth`** | `unsloth --job-json` | SFT / LoRA | Same as automodel — either runtime, 4 steps (download → train → upload → model-entity). Single-GPU inside the container | User asks for Unsloth, or wants Unsloth's 4-bit LoRA path / optimizer defaults on a single GPU | | **`rl`** | `rl --job-json` | **DPO** (preference) or **GRPO** (Gym env) | **Kubernetes only** — provisions a **Ray** cluster, no Docker fallback; 4 steps (download → train → upload → model-entity) | Preference DPO (full-weight), or GRPO with an environment FileSet (`native-v1` / `wheels-v1` / `adapter-wheels-v1`) + Gym rollout rows; GRPO also supports LoRA | `nemo-customizer` is the router (`nemo customization …`); training backends are separate plugins (`nemo-automodel`, `nemo-unsloth`, `nemo-rl`). The backend command posts to the platform API; the platform runs training in container steps — **not** in the CLI shell. Heavy ML deps live in container images only. **Runtime split:** `automodel`/`unsloth` run on **either** `platform.runtime` — Kubernetes GPU pods or the local Docker GPU executor (`require_container_runtime`), whichever the platform offers. Their one exception is **multi-node** (`parallelism.num_nodes > 1`), which compiles to a `gpu_distributed` executor only Volcano/Kubernetes can place. `rl` is Kubernetes-only in every case (`require_distributed_runtime`) — it provisions a Ray cluster and has no Docker fallback. So a docker-runtime platform runs SFT/LoRA fine and cannot run DPO or GRPO at all. Decision rule below in **Plugin pick**. Batch shell work; reuse resources with `--exist-ok`; skip CLI `--help` unless a command fails. ## Pre-flight — CLI resolution Run from the **nemo-helix** git root (top-level `pyproject.toml`), not a plugin subfolder. Example commands below use `nemo …` — resolve the invocation **once** before any other step: ```bash cd /path/to/nemo-helix if command -v nemo >/dev/null 2>&1; then echo "nemo" elif command -v uv >/dev/null 2>&1 && uv run nemo --help >/dev/null 2>&1; then echo "uv run nemo" else echo "CLI_NOT_FOUND" fi ``` | Result | Action | |--------|--------| | `nemo` | Use `nemo …` for all commands in this workflow | | `uv run nemo` | Prefix every command with `uv run` (repo dev checkout without `nemo` on `PATH`) | | `CLI_NOT_FOUND` | Stop. Route to **nemo-setup** (`make bootstrap` then `nemo setup` from the nemo-helix repo root). Do not continue. | ## Authentication (optional) Platform auth is **not required** to run customization when the cluster has authentication disabled. Check with `nemo auth status` — if it reports authentication is disabled, skip login and proceed. When auth **is** enabled on the connected platform, API calls need credentials: | Situation | Action | |-----------|--------| | Auth disabled | Skip login | | Auth enabled, unsigned JWT allowed (typical local dev: `auth.allow_unsigned_jwt: true`) | `nemo auth login --unsigned-token --email <user email or admin@example.com>` | | Auth enabled, OIDC configured | `nemo auth login` (or `--username` / `--password` for non-interactive) | | 401/403 on any platform call | Run the matching login above, then retry | Use `admin@example.com` unless the user specifies another email. Run `nemo auth status` after login to confirm. ## HuggingFace token (gated models) Gated HF repos need a platform secret (convention: **`hf-token`**) referenced as **`token_secret`** on the **model fileset** — not in job JSON (unlike W&B's `api_key_secret`). The Files service does **not** read your local `~/.cache/huggingface` or shell `HF_TOKEN`. | Model access | Action | |--------------|--------| | Public (e.g. `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16`) | Skip; omit `token_secret` on the fileset | | Gated / private HF repo | Before model fileset creation or job submit: `nemo secrets list --workspace default` and confirm `hf-token` exists. If missing, **ask the user** for their HF token and **stop** — do not create the fileset or submit until wired up. | Full create/update commands, fileset `token_secret`, license acceptance, and download-phase errors: `references/troubleshooting.md` § **Gated HuggingFace models**. ## Plugin pick 1. Run `nemo jobs list-execution-profiles -f json` (login first only if auth is enabled — see **Authentication**; see `references/troubleshooting.md` for parsing). 2. If the task is **DPO / preference optimization** (a `{prompt, chosen, rejected}` dataset, "align", "preference", "RLHF-style"), **GRPO / Gym / Prime Intellect env**, **or** the user explicitly asked for NeMo-RL → **`rl`** (requires a GPU profile **and** `platform.runtime: kubernetes`). 3. Else if the user explicitly asked for Unsloth → **`unsloth`**. 4. Else if the user explicitly asked for Automodel → **`automodel`**. 5. Else if any profile has `provider: gpu` or `gpu_distributed` → **`automodel`** (default, SFT/LoRA). 6. Else stop and tell the user GPU customization is unavailable (all backends need a GPU execution profile; `automodel`/`unsloth` accept either runtime — except multi-node, which needs Kubernetes — while `rl` needs `platform.runtime: kubernetes`). **`rl` runtime gate:** `rl --job-json` fails fast unless the platform runs `platform.runtime: kubernetes` (`require_distributed_runtime`). rl job steps execute as **Kubernetes pods via the `kubernetes_job` execution backend** — the **`docker` job backend cannot run rl**. Before submitting rl, confirm with `nemo jobs list-execution-profiles -f json` that the `cpu`/`gpu` profiles report `backend: kubernetes_job` (or `volcano_job`). If they report `backend: docker`/`subprocess`, the platform is **not** configured for rl: stop and tell the user DPO and GRPO need a Kubernetes-runtime platform — do **not** start/reuse a docker-runtime platform, and do **not** fall back to automodel/unsloth (those are SFT/LoRA, neither DPO nor GRPO). To stand up or configure one, see `references/rl-kubernetes-runtime.md`. For **`automodel`/`unsloth`**, training never runs inside the `nemo` CLI process. After submission, the platform launches GPU container steps — as Kubernetes pods, or on the Docker daemon attached to that platform host when the platform is Docker-backed. On a Docker-backed platform that daemon is often the same machine as `http://127.0.0.1:8080`, but always query the platform for its executors — not the agent's shell GPU or a separate `docker info` on another box. **`rl` does not use the Docker executor** — its steps run on the Kubernetes cluster the platform is configured against. ## Gotchas - Resolve the CLI per **Pre-flight — CLI resolution** before any `nemo …` command; run from the **nemo-helix** git root, not a plugin subfolder. - Set `NHX_BASE_URL` only when the user gives a platform URL; default `http://127.0.0.1:8080` (same as `http://localhost:8080`). The `nemo` CLI reads this env var (see SDK `NHX_BASE_URL`). Track whether the user **overrode** the base URL — see **Platform unreachable** below. - **Platform unreachable** — if any platform API call fails with a connection error (`Connection error`, timeout, refused): - **User gave a custom URL** or you exported a non-default `NHX_BASE_URL`: stop and tell the user the platform is not reachable at that address. Do **not** offer to start local services. - **Default URL only** (no user override): **ask** whether to start the platform locally. If they agree, from the **nemo-helix** git root run in the **background**: ```bash nemo services run \ --host 0.0.0.0 \ --port 8080 \ --controllers jobs,entities,models \ --service-group all ``` Poll until healthy (`curl -sf http://127.0.0.1:8080/health/ready` or retry `nemo jobs list-execution-profiles -f json`), then continue the workflow. Do not start services without asking. - ⚠️ **This default start is a DOCKER-runtime platform — valid for single-node `automodel`/`unsloth` only.** It is **NOT** valid for **`rl`**: rl needs `platform.runtime: kubernetes` with a `kubernetes_job` execution backend. Starting this default and submitting rl will fail the runtime gate. For rl, configure/point at a Kubernetes-runtime platform instead — see `references/rl-kubernetes-runtime.md`. Never start or reuse a docker-runtime platform for rl. - **Creating the model and dataset in one step** — instead of the separate fileset/upload/model-entity commands, use `--upload-model <local path or HF repo id>` and `--upload-dataset <local path>`; `rl` GRPO also has `--upload-environment <local dir>`. Any flag works on its own. Two surfaces: - `nemo customization --upload-model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 --upload-dataset ./sft-data` creates the resources and prints the refs as JSON on stdout, to paste into the job JSON. - `nemo customization <plugin> --job-json job.json --upload-model … --upload-dataset …` does the same and fills the refs into the job it submits. The job JSON file on disk is **not** rewritten, and `model`/`dataset` may be **left out of the file entirely** when the matching flag is passed. A local source is **one file or one directory**, exactly like `nemo files upload`; a directory is uploaded recursively. `--upload-model` also takes a HuggingFace repo id instead of a local path. For several files in one fileset, pass the directory holding them (`train.jsonl` + `val.jsonl` for automodel, `training.jsonl` + `validation.jsonl` for rl). Uploaded files keep their local names, and automodel's `train*`/`val*` discovery then finds both inside the one fileset. Fileset names are derived from the source (`nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16` -> `nvidia-nemotron-3-5-lightning-30b-a3b-bf16`, `sft-data/` -> `sft-data`); a name the platform would reject gets a `model-`/`dataset-` prefix (`2024-train.jsonl` -> `dataset-2024-train`). - **A reference belongs in one place only** — submit **fails** when the job JSON already sets `model` or `dataset` and you pass the matching `--upload-*` flag, because the two name different resources and picking one silently would train on the wrong input. Remove the field from the job JSON to create it at submit time, or drop the flag to use the ref the file names. It refuses before creating anything. - **One `--upload-dataset` fills both dataset fields** — automodel gets `dataset.training` **and** `dataset.validation`, unsloth gets `dataset.path` **and** `dataset.validation_path`, both pointing at the one fileset. Upload a **directory** holding both splits; automodel's `train*`/`val*` discovery then picks them apart inside it. `rl` has a single `dataset` ref and needs `training.jsonl` + `validation.jsonl` in that directory. - **`--upload-environment` uploads a Gym package, it does not build one** — validate it first with `pi-to-gym-conversion --validate-only <dir>` from the dedicated packaging env, not `uv run` (`references/gym-environments.md`). The flag is rejected on `automodel` and `unsloth`, which have no environment. - **`--upload-dataset` takes local files only** — converting an HF dataset into the backend's format is still a separate step you do first. Passing an HF dataset id fails. - **⚠️ `--exist-ok` reuses a fileset WITHOUT re-uploading its files** — this matches the Files service, and it is the most likely way to train on the wrong data. If the fileset already exists, `--upload-dataset ./data --exist-ok` keeps whatever was uploaded the first time; editing `train.jsonl` locally and re-running trains on the **old** contents, with no error. The CLI prints `Reused <ref>, which already existed.` and a note that files were not refreshed — read it. When the data has changed, create a **new** fileset by default: give the local source a new name (for example `sft-data-v2/`) and re-run without `--exist-ok`, so the job references a fileset holding exactly the new data. Do not upload into the existing fileset by default: `nemo files upload` adds and overwrites files but never removes ones renamed or deleted locally, and every other job or model entity referencing that fileset would see the changed contents. Update it in place only after the user confirms nothing else uses it and has checked its current files (`nemo files list <name> --workspace <workspace>`), then run `nemo files upload <path> <name> --workspace <workspace>`. For the same reason, **do not delete it on your own**: if deleting really is the right fix, show the user the exact `<workspace>/<name>` and run `nemo files filesets delete <name> --workspace <workspace>` only after they explicitly confirm. Always pass the workspace the user confirmed: the `nemo files` commands otherwise use the active CLI workspace, which can hold a different fileset with the same name. Without `--exist-ok`, the command fails when the resource exists, which is the safe default. One `--exist-ok` applies to whichever sources were passed. - **All backends run through the Jobs service** — use `nemo customization <plugin> --job-json …`; automodel, unsloth, and rl expose no local `run` verb. Do not improvise verbs or pass `--venv`. - **Historical jobs are on `/jobs`** — `GET /apis/customization/v2/workspaces/{workspace}/automodel/jobs`, `/unsloth/jobs`, and `/rl/jobs` return jobs whose spec stores `backend`. To list jobs submitted before that field existed, use `nemo jobs list` (`GET /apis/jobs/v2/workspaces/{workspace}/jobs`). - **Test fixtures are not the schema.** `tests/fixtures/*.json` are smoke-test inputs: they carry whatever made a test cheap, exercise one path rather than the field set, and nothing fails when the schema gains a field they never set. Read one for where a block sits in the payload — never for which fields exist, what a default is, or what a sensible value looks like, and never conclude a field is unsupported because a fixture omits it. Authoritative, in order: `nemo customization <plugin> explain` (the installed build's live schema), then the schema source files in `references/hyperparameters.md` § **Source of truth**, then this skill. When a fixture and `explain` disagree, the fixture is stale — say so rather than following it. - **Never set `max_steps` together with `epochs`** (automodel + unsloth; rl has the same caveat — see **rl (DPO / GRPO) gotchas**). `max_steps` is a global cap and stops mid-epoch. Every fixture in this repo sets it so a smoke test finishes in a minute — the most-copied wrong value here. Unsloth's schema enforces this as a hard mutex; automodel allows both but the result is surprising. - **Job done (all backends) = top-level `status`** in `completed` | `error` | `cancelled`. Steps can all be `completed` while the job is still `active` (upload, entity registration). `status_details.phase` may stay `training` with `progress_pct: 100` for a long time — keep polling. `poll_customization_job.sh` works for any job id (`automodel-…`, `unsloth-…`, or `rl-…`); it exits **1** on `error` or `cancelled`. - Model spec fills async: **submit without polling** `nemo models get` unless submit fails. - HF dataset id from the user → convert locally; do not ask for local paths first. - Dataset fileset name = HF dataset **name** only (`tau/commonsense_qa` → `commonsense_qa`), not the model name.