// Cherry-pick merged PRs labeled for a release branch into that branch, then open a PR and apply the cherry-pick-done label. Use when asked to "cherry-pick PRs for release/X.Y.Z", "pick PRs to release branch", or "cherry-pick labeled PRs".
Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL). Triggers on "evaluate model", "benchmark accuracy", "run MMLU", "evaluate quantized model", "run nel". Handles deployment, config generation, and evaluation execution. Not for quantizing models (use ptq), deploying/serving models (use deployment), or comparing completed baseline-vs-quantized results (use compare-results).
Establish baseline-vs-candidate evaluation plans, delegate missing evaluations, compare validated results, and decide quantization feasibility. Use when the user asks to compare baseline vs quantized runs, explain an accuracy drop/regression, verify whether a quantized checkpoint is acceptable, or compare NEL/MLflow evaluation outputs. Do NOT use for generic single-model evaluation without comparison intent (use evaluation), live NEL status/debugging (use launching-evals), or generic MLflow browsing without a comparison goal (use accessing-mlflow).
Serve a quantized or unquantized LLM checkpoint as an OpenAI-compatible API endpoint using vLLM, SGLang, or TRT-LLM. Use when user says "deploy model", "serve model", "start vLLM server", "launch SGLang", "TRT-LLM deploy", "AutoDeploy", "benchmark throughput", "serve checkpoint", or needs an inference endpoint from a HuggingFace or ModelOpt-quantized checkpoint. Do NOT use for quantizing models (use ptq) or evaluating accuracy (use evaluation).
Run, monitor, analyze, and debug LLM evaluations via nemo-evaluator-launcher. Covers running evaluations, checking status and live progress, debugging failed runs, exporting artifacts and logs, and analyzing results. ALWAYS triggers on mentions of running evaluations, checking progress, debugging failed evals, analyzing or analysing runs or results, run directories or artifact paths on clusters, Slurm job issues, invocation IDs, or inspecting logs (client logs, server logs, SSH to cluster, tail logs, grep logs). Do NOT use for creating or modifying evaluation configs.
Monitor submitted jobs (PTQ, evaluation, deployment) on SLURM clusters. Use when the user asks "check job status", "is my job done", "monitor my evaluation", "what's the status of the PTQ", "check on job <slurm_job_id>", or after any skill submits a long-running job. Also triggers on "nel status", "squeue", or any request to check progress of a previously submitted job.
This skill should be used when the user asks to "quantize a model", "run PTQ", "post-training quantization", "NVFP4 quantization", "FP8 quantization", "INT8 quantization", "INT4 AWQ", "quantize LLM", "quantize MoE", "quantize VLM", or needs to produce a quantized HuggingFace or TensorRT-LLM checkpoint from a pretrained model using ModelOpt.
| name | release-cherry-pick |
| description | Cherry-pick merged PRs labeled for a release branch into that branch, then open a PR and apply the cherry-pick-done label. Use when asked to "cherry-pick PRs for release/X.Y.Z", "pick PRs to release branch", or "cherry-pick labeled PRs". |
Cherry-pick all merged main PRs labeled cherry-pick-<version> (but not cherry-pick-done) into the corresponding release/<version> branch, one by one in merge order.
Ask the user for the release version (e.g. 0.44.0) if not already provided.
Set VERSION=<version> for use in subsequent steps.
Use the GitHub search API to list PRs that have the cherry-pick label but not cherry-pick-done, sorted by merge date ascending:
gh api "search/issues?q=repo:NVIDIA/Model-Optimizer+is:pr+is:merged+base:main+label:cherry-pick-<VERSION>+-label:cherry-pick-done&sort=updated&order=asc&per_page=50" \
--jq '.items[] | [.number, .title, .pull_request.merged_at] | @tsv' \
| sort -t$'\t' -k3
Present the list to the user before proceeding.
Check out release/<VERSION>, creating it from the remote if it doesn't exist locally:
git fetch origin release/<VERSION>
git checkout release/<VERSION>
All PRs are squash-merged, so each has a single-parent commit. Retrieve the SHA for each PR:
gh pr view <NUM> --repo NVIDIA/Model-Optimizer --json mergeCommit --jq '.mergeCommit.oid'
Cherry-pick each commit with -s (DCO sign-off). GPG signing is handled automatically by the repo's git config.
git cherry-pick -s <SHA>
On conflict: Tell the user which PR caused the conflict and ask them to fix it, then continue:
git cherry-pick --continue
Push the cherry-picks to a new branch and open a PR targeting release/<VERSION>. The PR title lists every cherry-picked PR number. The body uses ## Cherry-picked PRs as the only heading with one - #<NUM> bullet per PR — no titles, no links, no extra text.
git checkout -B cherry-picks/release-<VERSION>
git push -u origin cherry-picks/release-<VERSION>
gh pr create \
--title "[Cherry-pick] PRs #<NUM1> #<NUM2> ..." \
--base release/<VERSION> \
--head cherry-picks/release-<VERSION> \
--body "$(cat <<'EOF'
## Cherry-picked PRs
- #<NUM1>
- #<NUM2>
...
EOF
)"
Add the cherry-pick-done label to every PR that was successfully cherry-picked:
for pr in <NUM1> <NUM2> ...; do
gh pr edit $pr --repo NVIDIA/Model-Optimizer --add-label "cherry-pick-done"
done