evaluation

## NeMo Evaluator Launcher Assistant Guide the user through creating NEL YAML configs, running evaluations, and monitoring progress. NEL manages serving for canaries and full runs; see [Step 3's serving-handoff check](references/launcher-workflow.md#step-3--configure-deployment) for external endpoints. ### Completed-run analysis For completed runs, skip launch/config; use launching-evals for artifacts/analysis and apply [run-validation.md](references/run-validation.md) before scores. Its policy overrides blanket uncapping and fixed token/context limits. ### Workspace integration If `MODELOPT_WORKSPACE_ROOT` is set, use the common skill's `workspace-management.md` and reuse existing workspaces (PTQ → checkpoint smoke test → NEL evaluation; carry compatibility requirements into NEL's deployment config). ### Workflow ```text - [ ] Step 0: Check workspace (if MODELOPT_WORKSPACE_ROOT set) - [ ] Step 1: Check `nel` install + existing config; set up `.env` (+ `modelopttools:eval-config` for judge-scored runs) - [ ] Step 2: Build base config (5-question flow OR shortcut) - [ ] Step 3: Configure deployment (model path, params, cross-check) - [ ] Step 4: Fill remaining ??? values - [ ] Step 5: Confirm tasks (iterative) - [ ] Step 6: Multi-node (if needed) - [ ] Step 7: Interceptors (if needed) - [ ] Step 7.5: Container auth (SLURM private images) - [ ] Step 8: Dry-run → canary → full run - [ ] Step 9: Verify completed run ``` --- ### nel-next path (Terminal-Bench 2.x, SWE-bench, …) — branch here FIRST A few **agentic** AA benchmarks do **not** run on the currently validated `nemo-evaluator-launcher` 0.2.6 path (Steps 1–9 don't apply). They run on **nel-next** (`nemo-evaluator[harbor]` 0.4.x) — a separate package, CLI (`nel eval run`), `-O` overrides, and `services`/`benchmarks`/`cluster`/`output` schema. If the user asks for one, do **not** add it to a 0.2.6 `evaluation.tasks` list — instead: 1. Read **`references/nel-next.md`** (shared: venv, schema, AWS creds, architecture, timeout strategy, MLflow, run flow) + the per-benchmark recipe `recipes/tasks/aa_next/{terminal_bench_2_1,swebench_verified}.md`; start from `recipes/examples/example_eval_next.yaml`. 2. Isolated nel-next venv: `"$SKILL_DIR/scripts/nel-next.sh" --setup-only` (keeps 0.2.6 `nel` untouched). 3. Run **`modelopttools:eval-config`** (Step 3b) to write the AWS-sandbox creds + harbor infra rows (`${NEL_NEXT_EVAL_IMAGE}`, `${HARBOR_*_ECR_REPOSITORY}`, `${HARBOR_ECS_REGION}`) into `.env`; always include the `output.export_config.mlflow` block. 4. Dry-run → canary → full (`nel-next.sh eval run`), then **push to MLflow** — SLURM doesn't auto-export, so run `nel-next.sh mlflow-push -r <run_id> -c <cfg>` after (config-driven; see `references/nel-next.md`). 5. Before reporting a full-run score, apply `references/run-validation.md`, including its benchmark-specific checks. Steps 1–9 below are currently validated with 0.2.6 — use them for everything else. --- ### MRCR (NeMo Gym `simple_agent`) path — branch here too MRCR **does** run on the currently validated 0.2.6 `nel` launcher (as a `nemo_gym` task, not nel-next), so Steps 1–9 apply — but it is mechanically special and **standalone** (one gym eval per config; never mix it with `aa/` tasks). It is simple as gym tasks go: `simple_agent`, **no judge** — deterministic prefix-gated grading, `HF_TOKEN` the only secret. **Not an AA benchmark** — never generate it for an "AA" request. If the user asks for MRCR: 1. Read **`references/gym.md`** (pinned launcher, gym prepare/reap machinery, pin↔container coupling, preflight gaps, failure modes) + **`recipes/tasks/gym/mrcr.md`**; start from **`recipes/examples/gym/example_mrcr.yaml`** (1M variant, like the golden). 2. **Pick the variant first** (`config_n3_1m` / `config_n3_128k` / `config`) — it sets the context cap, dataset *and* metric prefix; the three are not comparable; set it in **both** `data_prep_params` and `collect_rollout_params`. Take the largest variant within the checkpoint's trained context — see the recipe. 3. `.env`: `HF_TOKEN` (dataset + n3 tokenizer are gated) plus `NEMO_EVALUATOR_TRUST_PRE_CMD=1` (the `pre_cmd` installs `tiktoken` + `transformers`; prepare fails without it) and `NEMO_EVALUATOR_TRUST_UNLISTED_TASKS=1` (`nemo_gym` is not in the FDF map). 4. **MRCR needs a git-backed Gym image.** The pin is newer than any image's baked Gym and must apply, so the template's `container:` is `???` and the bootstrap exits 1 on a non-git `/opt/Gym` (the public `eval-factory/nemo-gym:*` images). NVIDIA-internal: `modelopttools:eval-config` Step 3d names a working image. 5. Long-context deploy: for **1M**, `--max-model-len 1100000` + `VLLM_ALLOW_LONG_MAX_MODEL_LEN=1`; for **128K**, `--max-model-len` at the checkpoint's trained context (room for the answer above 131,072) and no override. 1M also needs `gpu_memory_utilization: 0.95`. Both: multi-instance fan-out; **never cap output tokens**; report the needle-count strata alongside `pass@1/accuracy`. 6. Run both dry-run and launch through `"$SKILL_DIR/scripts/nel-gym.sh"`; it enforces the currently validated 0.2.6 launcher even if `nel` on PATH is stale and avoids an unset `NEL_INVOCATION_ID` failure before client startup. Canary with `limit_samples` — the MRCR template forwards it as `++limit` (see the recipe's Canary section); the prepare pass still runs in full. --- Detailed launcher instructions live in [launcher-workflow.md](references/launcher-workflow.md). Read only the sections needed for the current stage; retain the dry-run → canary → full-run gates. Existing configs must pass Step 3's serving-handoff check before Step 8. Paths in that reference are relative to this skill directory. ### Step 1 — Prerequisites Read [Step 1](references/launcher-workflow.md#step-1--prerequisites). ### Step 2 — Build base config (when not using shortcut) Read [Step 2](references/launcher-workflow.md#step-2--build-base-config-when-not-using-shortcut). ### Step 3 — Configure deployment Read [Step 3](references/launcher-workflow.md#step-3--configure-deployment). ### Step 4 — Fill remaining ??? values Read [Step 4](references/launcher-workflow.md#step-4--fill-remaining--values). ### Step 5 — Confirm tasks (iterative) Read [Step 5](references/launcher-workflow.md#step-5--confirm-tasks-iterative). ### Step 6 — Multi-node Read [Step 6](references/launcher-workflow.md#step-6--multi-node). ### Step 7 — Interceptors Read [Step 7](references/launcher-workflow.md#step-7--interceptors). ### Step 7.5 — Container registry auth (SLURM private images only) Read [Step 7.5](references/launcher-workflow.md#step-75--container-registry-auth-slurm-private-images-only). ### Step 8 — Run evaluation (gated dry-run → canary → full) Read [Step 8](references/launcher-workflow.md#step-8--run-evaluation-gated-dry-run--canary--full). ### Step 9 — Verify completed run Apply [run-validation.md](references/run-validation.md)'s **Bounded Evaluation-Failure Policy** and **Timeout and Output-Limit Accounting** before reporting scores. For comparisons, also apply its **External Baseline Sanity Check**, then use `compare-results`. --- Issues: <https://github.com/NVIDIA-NeMo/Evaluator/issues> · <https://github.com/NVIDIA-NeMo/Evaluator/discussions>