Implements NemoInferenceMiddleware plugins for in-process inference request/response interception in IGW. Use when building a middleware plugin, implementing process_request or process_response, handling MiddlewareCall config (inline or config_id), exposing config entity CRUD APIs, or wiring up the nemo.inference_middleware entry-point. Trigger keywords: inference middleware, NemoInferenceMiddleware, process_request, process_response, MiddlewareCall, config_id, ImmediateResponse, VirtualModel middleware, middleware plugin.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
Implements NemoInferenceMiddleware plugins for in-process inference request/response interception in IGW. Use when building a middleware plugin, implementing process_request or process_response, handling MiddlewareCall config (inline or config_id), exposing config entity CRUD APIs, or wiring up the nemo.inference_middleware entry-point. Trigger keywords: inference middleware, NemoInferenceMiddleware, process_request, process_response, MiddlewareCall, config_id, ImmediateResponse, VirtualModel middleware, middleware plugin.
The CRUD API follows the standard NeMo Platform workspace-scoped pattern
(POST /v2/workspaces/{workspace}/my-plugin-configs, etc.).
See plugin-service skill for the full CRUD pattern.
Separation of types:NemoEntity inherits workspace: str (required,
no default) from EntityBase. Inline config dicts (from MiddlewareCall.config)
contain only domain fields and have no workspace, so
MyPluginConfig.model_validate({"threshold": 0.8}) raises a validation error.
A separate MyConfigData(BaseModel) with only the domain fields works for both
the inline path and the entity store path — have validate_middleware_config
always return that type.
Cache accessor reference
Available from on_startup() onward:
# Model entitiesself.list_model_entities_for_workspace() # all workspacesself.list_model_entities_for_workspace("default") # filtered
entity = self.get_model_entity("default/llama-3b") # ModelEntity | None
providers = self.get_model_providers_for_model("default/llama-3b") # list[ModelProvider]# Resolve backend URL + served model name for a direct call
target = self.get_inference_url_and_model("default/llama-3b")
# target.model_provider_gateway_url → "http://nim-svc:8080/v1"# target.served_model_name → "meta/llama-3.2-3b-instruct"# VirtualModels
vm = self.get_virtual_model("default/my-alias") # VirtualModel | Noneself.list_virtual_models_for_workspace("default") # list[str]
Request and response context
request.typed_body
IGW populates request.typed_body with a TypedDict-validated view of the body
for known paths (v1/chat/completions, v1/messages, v1/responses). All three
SDK param types are TypedDicts — plain dicts at runtime. typed_body is
interchangeable with body; use request.path for format dispatch, not
isinstance.
# Use typed_body when available, fall back to raw body
body = request.typed_body if request.typed_body isnotNoneelse request.body
# In a response hook, read the original pre-middleware request:
original_body = ctx.original_request.typed_body # or ctx.original_request.body
ctx.original_request is captured before any request middleware runs. Its
typed_body and .body always reflect what the caller originally sent, even
after downstream middleware has mutated the live request.
response.typed_body and annotations
response.typed_body holds the SDK-native parsed response object
(ChatCompletion, Message) when IGW can parse the backend payload. For
non-streaming responses, if non-None, it is canonical — mutate it instead of
response.result.
Use this response contract:
Goal
How
Mutate an existing payload field (e.g. redact PII in choices[0].message.content)
Mutate typed_body when available, or result when no typed view exists
Add a new field to the response body (e.g. guardrails)
Request middleware can also annotate the eventual backend response, even though
the InferenceResponse object does not exist yet. Put those annotations in
ctx.response_body_annotations as a staging area:
When IGW later receives the backend response, it builds an InferenceResponse
and copies the staged values into response.response_body_annotations. From
that point on, response.response_body_annotations is canonical because it is
attached to the response being returned. Response middleware should preserve,
replace, or remove annotations there, not on ctx. Final serialization injects
only response.response_body_annotations into the response body.
response_body_annotations is currently only supported for non-streaming responses.
IGW accumulates annotations on streaming responses, but does not serialize
them into the returned SSE chunks yet.
RequestResult return values
Return
Effect
InferenceRequest
IGW resolves request.body["model"] to a provider and proxies
To mock get_middleware_config (entity store fetch), patch at the entity client level:
with patch("nemo_platform_plugin.entity_client.NemoEntitiesClient", return_value=mock_client):
result = await plugin.get_middleware_config("my_plugin_config", "ws/cfg")
Reference implementation
plugins/example-plugin/ contains a complete working example:
