inference

# Inference Reference — ModelProvider + VirtualModel + Switchyard ## Prerequisites This skill assumes the platform is already running and any required plugins (`nemo-switchyard`, `nemo-guardrails`) are loaded. For local-platform startup, Switchyard install, and DB-reset choices, follow the **setup playbook** (`SETUP.md` at the repo root) first and then return here. ## API key environment variables Applies to both branches. Before creating secrets or running notebook/test harness flows, confirm API key environment variables: 1. Check whether `INFERENCE_NVIDIA_API_KEY` and/or `NVIDIA_API_KEY` are already set in the user's shell. 2. Treat `INFERENCE_NVIDIA_API_KEY` as the source of truth for the `https://inference-api.nvidia.com/v1` provider secret and notebook/test harness flows. 3. If `INFERENCE_NVIDIA_API_KEY` is not set, ask whether the user has the same token in another environment variable. Common alias: `NVIDIA_INFERENCE_API_KEY`. 4. If the user identifies an alias variable, export it into `INFERENCE_NVIDIA_API_KEY` for the current shell/session before continuing. 5. If `NVIDIA_API_KEY` is required for a Build/NIM-style provider or seeding path such as `https://integrate.api.nvidia.com`, and is not set, ask whether another env var already contains that token and export it into `NVIDIA_API_KEY` if the user confirms. Example alias normalization: ```bash export INFERENCE_NVIDIA_API_KEY="$NVIDIA_INFERENCE_API_KEY" ``` Use `INFERENCE_NVIDIA_API_KEY` for inference-api secret creation: ```bash printf '%s' "$INFERENCE_NVIDIA_API_KEY" | nemo secrets create nvidia-inference-key \ --from-file - --workspace my-workspace ``` ## Environment - **Port**: `8080` (CLI default — do NOT pass a custom `--base-url`) - **`export NMP_BASE_URL=http://localhost:8080` — required when targeting a local platform.** If your `~/.config/nmp/config.yaml` already points at a remote cluster, the CLI uses that base URL and ignores the local platform entirely. Setting this env var overrides the config file for the current shell session. - Workspace: `my-workspace` (substitute as needed; `default` also works) - Backend: `https://inference-api.nvidia.com/v1` For platform startup (`nemo services run`), Switchyard install, and state reset, see the setup playbook (`SETUP.md` at the repo root). --- ## CLI gotchas (real failures observed) - **`nemo secrets create`** uses `--from-file` (pipe key in). No `--value` flag. - **`nemo inference providers create`** takes `<name>` as a **positional** arg. Same for `update-status`, `get`, `delete`. - **`nemo inference virtual-models create`** takes `<name>` as positional. - **There is no `nemo inference chat completions create` command.** Use `nemo inference gateway model post <path> <vm-name> --workspace <ws> --body '<json>'`. - **`example` is not a valid `--services` arg.** Valid services: `audit`, `auth`, `customization`, `data-designer`, `entities`, `evaluation`, `files`, `guardrails`, `hello-world`, `inference-gateway`, `intake`, `jobs`, `models`, `safe-synthesizer`, `secrets`, `studio`. - **The reconciler re-syncs `served_models` from backend discovery every few seconds and drops manually-registered entries.** Always point VM `models` at auto-discovered entity IDs (e.g. `my-workspace/aws-anthropic-claude-opus-4-5`) — these survive reconciler cycles. If a VM 404s shortly after working, the reconciler overwrote your entry. Re-run `update-status` with the **full list** (it replaces, not appends) or switch to auto-discovered IDs. - **`update-status` replaces the entire `served_models` list** — include all entries, not just the new one. --- ## Step 1 — Workspace + Secret + Provider `my-workspace` is not created automatically. Create it first, then the secret and provider. ```bash nemo workspaces create my-workspace printf '%s' "$INFERENCE_NVIDIA_API_KEY" | nemo secrets create nvidia-inference-key \ --from-file - --workspace my-workspace nemo inference providers create nvidia-inference \ --workspace my-workspace \ --host-url "https://inference-api.nvidia.com/v1" \ --api-key-secret-name "nvidia-inference-key" nemo wait inference provider nvidia-inference --workspace my-workspace ``` Auto-discovery runs within ~3s. For NVIDIA Build / NIM via `https://integrate.api.nvidia.com`, use `NVIDIA_API_KEY`: ```bash printf '%s' "$NVIDIA_API_KEY" | nemo secrets create nvidia-build-key \ --from-file - --workspace my-workspace nemo inference providers create nvidia-build \ --workspace my-workspace \ --host-url "https://integrate.api.nvidia.com" \ --api-key-secret-name "nvidia-build-key" nemo wait inference provider nvidia-build --workspace my-workspace ``` For Anthropic direct (needs auth header rewrite): ```bash printf '%s' "$ANTHROPIC_API_KEY" | nemo secrets create anthropic-api-key \ --from-file - --workspace my-workspace nemo inference providers create anthropic \ --workspace my-workspace \ --host-url "https://api.anthropic.com" \ --api-key-secret-name "anthropic-api-key" \ --auth-header-format "X-Api-Key: {{ auth_secret }}" \ --default-extra-headers '{"anthropic-version": "2023-06-01"}' ``` `{{ auth_secret }}` is Jinja2 — substituted at request time. Without it the gateway defaults to `Authorization: Bearer` which Anthropic rejects. --- ## Step 2 — Discover available models After the provider is ready, `served_models` contains auto-discovered entries. **Always use these entity IDs** in VM `--models` — they survive the reconciler. ```bash # All entity IDs (use these in --models and inference body) nemo inference providers get nvidia-inference --workspace my-workspace \ --output-format json | jq -r '.served_models[].model_entity_id' # Side-by-side: served_model_name → model_entity_id nemo inference providers get nvidia-inference --workspace my-workspace \ --output-format json \ | jq -r '.served_models[] | "\(.served_model_name) → \(.model_entity_id)"' # Filter by vendor prefix nemo inference providers get nvidia-inference --workspace my-workspace \ --output-format json \ | jq -r '.served_models[] | select(.served_model_name | startswith("aws/anthropic")) | .model_entity_id' # Count nemo inference providers get nvidia-inference --workspace my-workspace \ --output-format json | jq '.served_models | length' ``` Entity ID normalization: slashes/dots → dashes, workspace prefix added. `aws/anthropic/claude-opus-4-5` → `my-workspace/aws-anthropic-claude-opus-4-5` --- ## Step 3 — Body + model entity convention `body["model"]` must be a **real backend entity ID** that IGW can resolve — not the VM name. The VM name is resolved via the URL path (`gateway model post v1/chat/completions <vm-name>`), not the body. Exception: VMs that **rewrite** `body["model"]` (random_routing, `ModelFormatLookupProcessor` in translate) — here the initial body model can be the VM name since the rewrite resolves it to the real entity. For VMs without a rewriting middleware: always send the real auto-discovered entity in the body. ### Common failure: HTTP 422 from chat completion If `nemo agents invoke …`, an `openai`-SDK client, or a `langchain-openai` client returns HTTP 422 in well under a second with no provider call made, the gateway rejected the request's `model` field as malformed. The most common cause is sending the **upstream catalog name** (e.g. `meta/llama-3.3-70b-instruct` — vendor slash, dotted version) instead of the **hyphenated entity ID** (`meta-llama-3-3-70b-instruct`). The entity ID is what `nemo models list` returns and what every gateway input expects. The slash-with-dots form is `served_model_name` — a human-display alias from the upstream provider's catalog. It appears in provider metadata and error messages, but is **never** a valid request field. (A `workspace/entity-id` prefix is fine — that's a workspace qualifier, not the upstream alias. The rejected form is specifically `vendor/upstream.dotted.name`.) Diagnose and recover: ```bash # 1. What entity IDs actually exist? nemo inference providers get nvidia-build --workspace default \ --output-format json \ | jq -r '.served_models[] | "\(.served_model_name) → \(.model_entity_id)"' # 2. If you set NEMO_DEFAULT_MODEL, confirm it's the hyphenated entity ID. echo "$NEMO_DEFAULT_MODEL" # 3. If a NAT-style deployed agent's resolved config carries the upstream # slash form in model_name, re-deploy with NEMO_DEFAULT_MODEL set to the # entity ID. The .config.llms.agent.model_name path is specific to NAT # workflows (e.g. the calculator-agent example); other workflow types store # the model elsewhere. nemo agents deployments list --workspace default \ | jq -r '.data[] | "\(.name) model=\(.config.llms.agent.model_name // "n/a")"' ``` For an OpenAI-SDK or LangChain client, pass the entity ID (with or without the `workspace/` prefix) as `model=`: ```python client.chat.completions.create( model="default/meta-llama-3-3-70b-instruct", # OR "meta-llama-3-3-70b-instruct" messages=[...], ) # NOT: model="meta/llama-3.3-70b-instruct" → 422 ``` This is intentional gateway behavior: accepting arbitrary slash-prefixed names would force IGW to guess provider attribution for every request instead of resolving it from a registered entity. --- ## Step 4 — VirtualModel patterns ### Random routing — same format (deterministic test: `strong_probability=1.0`) ```bash nemo inference virtual-models create vm-random-strong --workspace my-workspace \ --models '[ {"model":"my-workspace/nvidia-mistralai-mixtral-8x22b-instruct-v01","backend_format":"OPENAI_CHAT"}, {"model":"my-workspace/nvidia-qwen-qwen3-32b","backend_format":"OPENAI_CHAT"} ]' \ --request-middleware '[{"name":"nemo-switchyard","config_type":"random_routing","config":{ "strong":{"model":"my-workspace/nvidia-mistralai-mixtral-8x22b-instruct-v01"}, "weak":{"model":"my-workspace/nvidia-qwen-qwen3-32b"}, "strong_probability":1.0, "rng_seed":42, "enable_stats":false }}]' ``` ### Random routing — cross format (chain routing + translate) `random_routing` picks the backend; `translate` rewrites the request format. Order matters: routing first, translate second. Use `response_middleware` too for full round-trip translation back to the client's format. ```bash nemo inference virtual-models create vm-random-cross --workspace my-workspace \ --models '[ {"model":"my-workspace/aws-anthropic-claude-opus-4-5","backend_format":"ANTHROPIC_MESSAGES"}, {"model":"my-workspace/nvidia-nvidia-nemotron-nano-31b-v3","backend_format":"OPENAI_CHAT"} ]' \ --request-middleware '[ {"name":"nemo-switchyard","config_type":"random_routing","config":{ "strong":{"model":"my-workspace/aws-anthropic-claude-opus-4-5"}, "weak":{"model":"my-workspace/nvidia-nvidia-nemotron-nano-31b-v3"}, "strong_probability":0.5, "enable_stats":false }}, {"name":"nemo-switchyard","config_type":"translate","config":{"target_format":"auto","enable_stats":false}} ]' \ --response-middleware '[{"name":"nemo-switchyard","config_type":"translate","config":{"target_format":"auto","enable_stats":false}}]' ``` ### Translate — cross format, full round-trip ✓ Client sends OpenAI shape, backend is Anthropic, response comes back as OpenAI. **Must list translate in BOTH `request_middleware` and `response_middleware`.** ```bash nemo inference virtual-models create vm-translate-cross --workspace my-workspace \ --models '[{"model":"my-workspace/aws-anthropic-claude-opus-4-5","backend_format":"ANTHROPIC_MESSAGES"}]' \ --request-middleware '[{"name":"nemo-switchyard","config_type":"translate","config":{"target_format":"anthropic","enable_stats":false}}]' \ --response-middleware '[{"name":"nemo-switchyard","config_type":"translate","config":{"target_format":"anthropic","enable_stats":false}}]' ``` ### Guardrails — adding content safety to a VirtualModel Guardrail rails are attached as a `nemo-guardrails` MiddlewareCall. Configs are created separately with `nemo guardrail configs create` — see the **nemo-guardrails** skill for rails config JSON, prompt templates, task LLMs, and streaming output rails. This section only covers the VirtualModel wiring.