agents-optimize

# NeMo Agent Optimization Optimize and accelerate a deployed agent by suggesting routing splits, prompt/hyperparameter tuning, and evaluations against newly available models. Always anchor proposed changes against an evaluation baseline so quality regressions are caught before promotion. ## Workflow 1. **Pick an agent.** Run `nemo agents list` and prompt the user to choose one to optimize. Use the rest of this skill on that single agent. 2. **Establish a baseline.** Look for prior evaluation results from `nemo agents evaluate`. If none exist but the agent has an eval dataset+config, run one and save the result. If no eval dataset exists, suggest the user create one — recent telemetry can usually be turned into a small dataset. 3. **Run the analysis steps below**, emit suggestions, and persist them. 4. **Present suggestions** to the user and let them pick which to apply. 5. **After applying any change**, re-run evaluation and compare to the baseline. Surface deltas (per-evaluator average score, wallclock, token cost) and confirm before promoting traffic. 6. **Apply via the create-sibling-and-deploy pattern.** Agents v2 has no PATCH on `/agents/<name>`, so model swaps and config edits go through: create a sibling agent → deploy it → run an eval against the sibling → delete old deployments so traffic flows to the new version. ## Analysis steps ### 1. Routing / switchyard split Inspect `agent.config.llms[*].model_name` from `nemo agents list`. If every LLM points at a single frontier model and **no** switchyard routing virtual model is in use, suggest creating a `random_routing` virtual model with an 80% strong / 20% weak split. - Pick the weak model from `nemo models list --filter.name nemotron` (prefer current Nemotron tiers, e.g. `nvidia-nemotron-nano-3b` or `nemotron-3-super`-class). Exclude content-safety / topic-control / safety-guard / GLiNER models — those aren't general chat replacements. - **Verify entity names before constructing the VM.** Auto-discovered names on the NVIDIA hub sometimes have doubled prefixes (e.g. `nvidia-nvidia-nemotron-3-super-v3`, not `nvidia-nemotron-3-super-v3`). Always confirm with `nemo -f json models get <candidate>` before referencing the name in `--models` or as the `weak`/`strong` model in middleware config. A 404 here, caught early, is much cheaper than a failed VM rollout. - Use **the `inference` skill** for the exact `nemo virtual-models create … --request-middleware … nemo-switchyard config_type=random_routing …` syntax (`strong_probability: 0.8`). - **After creating the VM, smoke-test routing before wiring an agent to it.** Run ~20 minimal `nemo inference gateway model post v1/chat/completions <vm>` calls and tally the response `.model` field. The split should approximate the configured probability (e.g. ~16/4 for 0.8/0.2 over 20 calls). If the split is 20/0, the middleware isn't actually rewriting the request — stop and investigate before deploying the sibling agent. Catches misconfigured probabilities, wrong entity IDs in middleware config, and other middleware- silently-bypassed failure modes. - Set `model` on the suggestion to the current source model. Suggested follow-up actions: `nemo agents evaluate --agent <agent> --eval-config <yaml>` against the routed VM before promoting. This is pure config inspection — no telemetry download required. ### 2. Prompt + hyperparameter tuning If `nemo agents optimize run-strategy` has not been run for this agent since the last snapshot, suggest running it. The job sweeps prompts and hyperparameters via `nat optimize` against the agent's eval dataset. Suggested actions: - `nemo agents optimize prepare-fileset --source <bundle-dir> --optimize-config <bundle-relative-yaml> --fileset <name>` - `nemo agents optimize run-strategy --strategy config-tuning --agent <name> --optimize-config-fileset <workspace/name> --optimize-config <config-path-in-fileset>` - After it completes, apply the new prompt + hyperparameters to a sibling agent, deploy it, and run `nemo agents evaluate` to compare. ### 3. New model scan Diff `nemo models list --all-pages` against `prevSnapshot.modelNames`. **Never emit on first run** — there is no prior snapshot to diff against. Emit one suggestion per new model whose parameter count and family fit the agent's task profile (skip guardrails / safety / GLiNER models). Each suggestion sets `model` to the new model name and includes: - Create a sibling agent pointing at `<new-model>`, then run `nemo agents evaluate --agent <sibling> --eval-config <yaml>` to compare against the current model on the baseline dataset. There is no `--model` override flag on `evaluate`; the model swap goes through the sibling agent, or by editing the eval-config YAML. - `nemo garak targets create <target> -d '{"model": "<new-model>", "type": "<type>"}'` then `nemo garak audit --spec '{"config": "default/<config>", "target": "default/<target>"}'` to verify the new model is robust against jailbreaks before promotion. Pure set subtraction — no downloads required. ## Persistence (every run) Uses fileset `nemo-agent-optimizer`. A downstream web hook also reads/writes this fileset; **changes here MUST stay backwards compatible** with `nemo-agents-optimize-sync-web` and the consumer code at `web/packages/studio/src/routes/AgentOptimizationsRoute/` (kept up to date via the parity skill above). | Path | Contents | |------|----------| | `optimizer_snapshot.json` | Per-agent snapshot map; one entry per agent ever audited | | `optimizer_suggestions.jsonl` | One JSON object per line; survives applies | ### Snapshot shape (per-agent, single file) The snapshot is **one JSON file at the fileset root**, keyed by agent name so running the skill against agent A never overwrites agent B's snapshot: ```json { "agents": { "support-bot": { "modelNames": ["..."], "agentNames": ["support-bot"], "updatedAt": "2026-05-07T12:34:56Z" }, "triage-bot": { "modelNames": ["..."], "agentNames": ["triage-bot"], "updatedAt": "2026-05-06T09:11:02Z" } } } ``` The legacy flat shape (`{ modelNames, agentNames, updatedAt }`) is **read-only backwards-compatible** — if you load a file without the top-level `agents` key, treat the whole object as the snapshot for the *current* selected agent and migrate to the keyed shape on the next save. Never write the legacy shape back out. ### Load → update → save flow 1. Try to load the prior snapshot (`nemo files download nemo-agent-optimizer --remote-path optimizer_snapshot.json -o …`). - 404 → no snapshot exists yet for the workspace; treat as first run. - Snapshot exists but `agents[<selected>]` is missing → first run *for that agent*. Do not emit `new_model_scan` suggestions (no prior model list to diff against). Other agents' entries are preserved as-is on save. - Snapshot exists and `agents[<selected>]` is present → use that entry's `modelNames` for the new-model-scan diff. 2. After analysis, **load → update only `agents[<selected>]` → save** the whole file. Never overwrite blindly with just the current agent's data — that would erase every other agent's snapshot. 3. **Merge** the suggestions JSONL (do not overwrite blindly): 1. Load the existing file. 2. Compute fresh suggestions for the selected agent. 3. Preserve every record where `applied === true` as-is. 4. Preserve every non-applied record whose `agent` field is **not** the currently selected agent — those belong to other agents' runs and must not be dropped here. (Workspace-scoped suggestions with no `agent` field are owned by the most recent run; replace them.) 5. Drop any fresh suggestion whose identity tuple `(type, agent, model)` matches an applied record. 6. Replace the selected agent's non-applied records with fresh output. 4. On first run the fileset may not exist — create it with `nemo files filesets create nemo-agent-optimizer`. ### JSONL record shape One JSON object per line. Required fields: `type`, `title`, `detail`. Optional: `agent`, `model`, `severity`, `suggested_actions`, `apply`, `apply_description`, `applied`, `applied_at`. Keep this backwards compatible — downstream web tooling consumes the fileset (see "Persistence" above). `detail` describes WHY the suggestion was raised. `apply_description` (if `apply` is set) is a short, concrete description of what executing the `apply` block would do (e.g. resource names that will be created). When a consumer of the fileset executes an `apply` block, it is expected to set `applied: true` and `applied_at: <ISO timestamp>` and re-upload the JSONL so the applied state survives across reads. | `type` value | Meaning | |--------------|---------| | `model_optimization` | Switchyard routing split or smaller-model fit | | `prompt_optimization` | `nemo agents optimize run-strategy` is recommended | | `new_model_scan` | New model appeared since last snapshot | `model_optimization` is reused for both "use a smaller model" and "add a routing split" so existing UI tiles render without a frontend change. The `title`/`detail` text describes which variant it is. ### Optional `apply` block (one-click action) When set, the `apply` block defines a single-action mutation that a downstream consumer can execute against the NeMo Helix API on the user's behalf. The contract is **strictly same-origin** to the NeMo Helix API host — consumers re-validate this at request time and reject anything else. Shape — single step (most common): ```json "apply": { "method": "POST" | "PUT" | "PATCH" | "DELETE", "path": "/apis/...", "body": { ... } } ``` Shape — multi-step (array, executed in order, stops on first failure): ```json "apply": [ { "method": "POST", "path": "/apis/...", "body": { ... } }, { "method": "POST", "path": "/apis/...", "body": { ... } } ] ``` Rules — these MUST be followed when emitting an `apply` block: - `method` must be one of `POST`, `PUT`, `PATCH`, `DELETE`. - `path` MUST start with `/`. It MUST NOT contain `://`, MUST NOT start with `//` (protocol-relative), MUST NOT contain query/fragment, and MUST NOT contain control characters. - The path is resolved against the configured NeMo Helix API URL; the frontend rejects any spec that resolves to a different origin. - The path's `workspaces/<ws>/` segment MUST equal the workspace the suggestion was emitted for. Cross-workspace mutations are rejected. - The `(method, pathname)` pair MUST be on the optimizer apply allowlist. Currently allowed: - `POST /apis/agents/v2/workspaces/<ws>/agents` — create sibling agent (`body.name` required) - `POST /apis/agents/v2/workspaces/<ws>/deployments` — deploy `suggestion.agent` OR a name declared by an earlier `POST /agents` step in this apply - `POST /apis/agents/v2/workspaces/<ws>/jobs/evaluate` — submit an evaluate-agent platform job. `body.spec.agent` must be a platform agent ref equal to `suggestion.agent` or a name declared by an earlier `POST /agents` step in the same apply. Endpoint URLs and cross-workspace refs are rejected. - In-place `PATCH /agents/<name>` is **not allowed** — Agents v2 only implements POST/GET/LIST/DELETE. Model swaps must go through the create-sibling-and-deploy pattern below. - Any other path is rejected at apply time. New action types require updating both this skill and the frontend allowlist in `web/packages/studio/src/routes/AgentOptimizationsRoute/api.ts`. - `body` must be JSON-serializable. Embed workspace/agent/model identifiers literally — do not template at apply time. If you cannot construct a safe, fully-qualified payload, omit `apply`. The textual `suggested_actions` remain available as a fallback. #### Example: routing/smaller-model swap Multi-step — create the sibling, then deploy it: ```json {"type":"model_optimization","title":"Consider smaller model for support-bot","detail":"Agent uses 70B; 4B Nemotron is available.","suggested_actions":["nemo agents evaluate ...","switchyard for automatic routing"],"apply":[{"method":"POST","path":"/apis/agents/v2/workspaces/default/agents","body":{"name":"support-bot-nemotron-mini-4b","config":{"llms":{"llm":{"model_name":"nvidia-nemotron-mini-4b"}}}}},{"method":"POST","path":"/apis/agents/v2/workspaces/default/deployments","body":{"agent":"support-bot-nemotron-mini-4b"}}]} ``` Three-step — create sibling, deploy, evaluate so quality is measurable before promoting: ```json {"type":"model_optimization","title":"Validate smaller-model swap for support-bot","detail":"Agent uses 70B; 4B Nemotron is available. The third step queues an evaluation against the sibling so quality is measurable before promoting it.","apply_description":"Creates 'support-bot-nemotron-mini-4b', deploys it, and submits an evaluate-agent job using the eval config in fileset 'support-bot-eval'.","suggested_actions":["nemo agents evaluate ...","switchyard for automatic routing"],"apply":[{"method":"POST","path":"/apis/agents/v2/workspaces/default/agents","body":{"name":"support-bot-nemotron-mini-4b","config":{"llms":{"llm":{"model_name":"nvidia-nemotron-mini-4b"}}}}},{"method":"POST","path":"/apis/agents/v2/workspaces/default/deployments","body":{"agent":"support-bot-nemotron-mini-4b"}},{"method":"POST","path":"/apis/agents/v2/workspaces/default/jobs/evaluate","body":{"spec":{"agent":"support-bot-nemotron-mini-4b","eval_config":"eval.yaml","eval_config_fileset":"support-bot-eval","output":"support-bot-eval-out"}}}]} ``` Notes when emitting the eval step: - `body.spec.agent` MUST be `suggestion.agent` or a name declared by an earlier `POST /agents` step in the same apply. Endpoint URLs (`http(s)://...`) are rejected. - `body.spec.eval_config` is interpreted relative to the downloaded fileset when `eval_config_fileset` is set. - `body.spec.workspace` is overwritten by the URL workspace at compile time, so it can be omitted. - Only emit the eval step when you have a concrete eval config + fileset to point at — half-configured eval submissions just create a failed job. ## CLI reference (verified) ```bash # List agents and models nemo agents list nemo models list --all-pages # always --all-pages; default paginates nemo models list --filter.name nemotron # find Nemotron candidates # Optimization commands. # The optimize command's --optimize-config value must be relative to the staged fileset root. nemo agents evaluate --agent <name> --eval-config <yaml> nemo agents optimize prepare-fileset --source <bundle-dir> --optimize-config <bundle-relative-yaml> --fileset <name>