cae-streaming

name	cae-streaming
description	Run Kit-CAE as a WebRTC-streamed application that remote clients can view and control via data-channel messages. Provides a streaming-enabled .kit dep set, the wire-protocol reference, the messaging-API reference, a small reusable Python helper (`OmniverseAPI`), and a runnable reference handler (`load_scene` + scenes registry) that demonstrates the request/response pattern end-to-end. Triggers on requests involving "stream Kit-CAE", "WebRTC", "remote viewer", "browser viewport", "application streaming", "data-channel messages", or "remote control of Kit-CAE".
depends	["cae-core"]
version	2.1.0
metadata	{"author":"NVIDIA","tags":["kit-cae","cae","streaming","webrtc","remote","data-channel","omniverse"]}

CAE Streaming - Remote-Controlled Kit-CAE

Stand up Kit-CAE as a streaming application that a remote client (browser, agent, dashboard, notebook) can both watch (live WebRTC viewport) and drive (request/response messages over the data channel).

The skill ships:

• A streaming-enabled .kit dep set (omni.kit.livestream.messaging + a hideUi setting) you can point any custom .kit at.
• scripts/launch_streaming.sh — wraps repo.sh launch with sensible defaults.
• scripts/omniverse_api.py — a small @api.request / @api.signal decorator framework that handles sync AND async handlers.
• scripts/serve.py — a runnable reference handler (load_scene + scene registry) you can study or replace.
• The wire protocol (references/streaming-protocol.md) and the underlying Python API (references/messaging-api.md).

load_scene is one example. Real consumers will write their own handlers.

Purpose

Run Kit-CAE as a WebRTC-streamed application that remote clients can view and control via data-channel messages. Provides the streaming-enabled .kit dep set, the wire-protocol and messaging-API references, a reusable Python helper (OmniverseAPI), and a runnable reference handler (load_scene + scenes registry).

Prerequisites

cae-core (loaded automatically). A built Kit-CAE checkout that includes omni.kit.livestream.messaging. See ## Standing up the streaming server for the launch command and references/streaming-protocol.md for the wire format.

Instructions

1. Stand up the streaming server — follow the launch command in ## Standing up the streaming server; scripts/launch_streaming.sh wraps repo.sh launch with the right defaults.
2. Run the reference handler — invoke scripts/serve.py with the run_script pattern (./repo.sh launch -n <kit> -- --exec scripts/serve.py) to host the bundled load_scene handler; see ## Bundled example: load_scene + scenes registry.
3. Connect a client — open the WebRTC viewer and connect over the data channel as described in ## Connecting a client.
4. Write your own handlers — apply the @api.request and @api.signal patterns in ## Writing handlers and the wire format in references/streaming-protocol.md.
5. Troubleshoot — consult ## Troubleshooting for common failures (port collisions, async coroutine handling, scene-id mismatches).

Examples

Bundled scripts:

Script	Purpose
scripts/launch_streaming.sh	Wrap repo.sh launch with sensible streaming defaults.
scripts/serve.py	Runnable reference handler — load_scene + scenes registry.
scripts/omniverse_api.py	@api.request / @api.signal decorator helper for sync and async handlers.

End-to-end runnable example: scripts/serve.py paired with the scenes registry in scenes.yaml. See ## Writing handlers for the request/response pattern and references/messaging-api.md for the full helper API.

Limitations

Streaming uses WebRTC — clients need a browser or WebRTC client. LivestreamMessaging is per-app, not per-session, so multiple clients see the same scene. Time-varying playback inherits the same constraints as cae-visualization.

When to use

• A user / agent needs to interact with Kit-CAE from a process that isn't Kit (browser, REST service, RPC client, ML pipeline, dashboard).
• You need a long-lived Kit-CAE session that responds to commands rather than one-shot scripts that exit.
• You're building any client UI that wants a live Kit viewport plus programmatic control.

For headless capture (clean PNG/EXR/MP4 outputs, no streaming), stay with cae-capture. This skill is the live + interactive path, not the offline render path.

Architecture

  Remote client (browser, agent, …)        Kit-CAE host
  ┌──────────────────────────────┐         ┌──────────────────────────────┐
  │ AppStreamer (WebRTC client)  │ video   │ omni.kit.livestream.app      │
  │  - <video> / <audio> sinks   │ ◄────── │ omni.kit.livestream.webrtc   │
  │                              │         │                              │
  │ data-channel:                │ JSON    │ omni.kit.livestream.messaging│
  │  - send  ({type}_request)    │ ◄────── │ scripts/serve.py:            │
  │  - recv  ({type}_response)   │ ──────► │   @api.request handlers      │
  │  - recv  ({signal})          │ ◄────── │   @api.signal pushes         │
  └──────────────────────────────┘         └──────────────────────────────┘

Standing up the streaming server

Minimal launch:

cd <kit-cae-dir>
bash skills/cae-streaming/scripts/launch_streaming.sh

What it does:

1. Launches a streaming Kit-CAE app. The launcher defaults to omni.cae_vtk_streaming.kit (streaming + VTK extensions). Override via CAE_STREAMING_KIT=<other.kit> for the slim non-VTK build (omni.cae_streaming.kit) or any other custom app.
2. Both shipping streaming apps depend on kit_cae_streaming (the streaming base template), which lists omni.kit.livestream.messaging and sets [settings.app.window] hideUi = true. The data-channel observe API and the chrome-free viewport are both wired in via that template — no special launch flags required.
3. Runs scripts/serve.py via --exec. That script registers handlers and loops forever, awaiting messages from connected clients.

After the launch, look for these log lines:

[ovapi] registered request handler: load_scene_request
[ovapi] registered request handler: list_scenes_request
[cae-streaming] viewport-only: hideUi=True via carb.settings
[omni.kit.livestream.app.plugin] Started primary stream server on signal port 49100 …

Now any compatible client can connect.

Connecting a client

Any client speaking omni.kit.livestream.*'s WebRTC + data channel works. Compatible clients include:

• A JavaScript browser app using @nvidia/omniverse-webrtc-streaming-library.
• A Python client over aiortc.
• A native Kit window pointed at the same signaling endpoint.

Whatever the language, the JSON wire format is the same — see references/streaming-protocol.md for the envelopes, naming convention, and correlation rules.

A minimal browser snippet (JS):

import { AppStreamer, StreamType } from "@nvidia/omniverse-webrtc-streaming-library";
await AppStreamer.connect({
  streamSource: StreamType.DIRECT,
  streamConfig: {
    signalingServer: "kit-host", signalingPort: 49100,
    videoElementId: "kit-video", audioElementId: "kit-audio",
    onCustomEvent: (msg) => console.log("from kit:", msg),
  },
});
AppStreamer.sendMessage(JSON.stringify({
  event_type: "load_scene_request",
  payload: { id: 1, scene_id: "static_mixer" },
}));

The wire protocol

Two message families:

• request / response — client asks Kit to do something; correlation by id field. Naming convention: <type>_request → <type>_response (underscore — see the colon-default warning in references/messaging-api.md).
• signal — one-way push from Kit to client; no id, no reply expected.

Full spec, including the JSON envelopes, the bundled-example operations, and the error encoding: references/streaming-protocol.md.

Writing handlers

Use scripts/omniverse_api.py's @api.request / @api.signal decorators (handles sync and async); drop down to omni.kit.livestream.messaging directly via references/messaging-api.md when you need finer control.

Request/response with OmniverseAPI

from omniverse_api import OmniverseAPI, exclusive
from omni.cae.core.commands import execute_command
from omni.cae.usd_plugins_importers import import_to_stage
from omni.cae.testing import wait_for_update

api = OmniverseAPI()

@api.request
@exclusive  # optional: drop concurrent calls if mutating shared state
async def load_scene(scene_id: str = "", **_):
    if not scene_id:
        return {"ok": False, "error": "scene_id required"}
    await import_to_stage(f"/data/{scene_id}.cgns", f"/World/{scene_id}")
    await wait_for_update()
    await execute_command("CreateCaeVizFaces",
                          dataset_path=f"/World/{scene_id}/Base/Zone",
                          prim_path=f"/World/CAE/Faces_{scene_id}")
    return {"ok": True, "scene_id": scene_id}

The decorator handles payload unpacking, async scheduling, response dispatch, and exception → {"ok": False, "error": "..."} conversion (so Python tracebacks never cross the wire).

Pushing signals

@api.signal
def progress(scene_id: str, percent: int):
    return {"scene_id": scene_id, "percent": percent}

# Calling progress(scene_id="static_mixer", percent=42) pushes a
# `progress_signal` event with that payload to all connected clients.

# Or fire a payload directly without a wrapper function:
api.dispatch_signal("notification", {"level": "info", "message": "Stage ready"})

Two footguns to avoid

• Long-lived serve.py. Do NOT use the cae-core "Script shutdown (MANDATORY)" template (app.post_quit() + os._exit(0)). That's for one-shot capture; a streaming listener must run until the process is signaled.
• Document new request types in references/streaming-protocol.md so future readers inherit the contract.

Bundled example: load_scene + scenes registry

The skill ships with one runnable handler so the end-to-end flow works the day you check it out:

• scripts/serve.py registers load_scene_request, list_scenes_request, and register_scene_request handlers.
• A scene registry maps scene_id → { path, format, default_viz, ... }. serve.py resolves it in this order: scenes.json (machine-friendly, no extra deps) → scenes.yaml (requires PyYAML) → an embedded DEFAULT_SCENES dict in serve.py itself (always works).

All three layers are illustrative — copy and adapt freely. The framework itself does not require a registry; it exists because the bundled handler reads it.

Field reference (when you do use the bundled handler):

Key	Meaning
path	Filesystem path. ${KIT_CAE_DIR} is expanded from the env var.
format	One of cgns, vtk, npz, ensight. Selects the importer.
prim_path	Where the data lands in the stage (e.g. /World/<name>).
dataset_path	Prim with the dataset (per-format; see cae-core/references/formats.md).
default_viz	Short label appended to the viz prim path.
default_viz_command	One of the CreateCaeViz* commands from cae-visualization.
color_field	Optional. Path to the field bound as the viz colors target.

Verify dataset paths with cae-core/scripts/inspect_cgns.py / inspect_vtk.py before declaring done — paths vary per file.

Ports, hosts, firewalls

• WebRTC signaling: 49100 (configurable via --/exts/omni.kit.livestream.app/primaryStream/streamPort=).
• Media negotiation occurs over ranges 47995-48012 and 49000-49007 by default.
• Open these ports between client and Kit host (or use a TURN server if you cannot punch through NAT).
• HTTPS / WSS: not required for local testing. For remote deployment, terminate TLS in front of the signaling endpoint.

Viewport-only streamed output

The streaming .kit template (templates/kit_cae/kit_cae_streaming.kit) sets [settings.app.window] hideUi = true, which collapses Kit's editor chrome (menubar, panels, toolbar, timeline) and leaves only the rendered viewport. serve.py defensively re-applies the same setting at startup via carb.settings.set("/app/window/hideUi", True) plus toggles for the in-viewport gizmos (camera/light/HUD/grid/axis). Override at launch with --/app/window/hideUi=false for local debugging.

Troubleshooting

Symptom	Likely cause
Client stuck "Connecting…"	Port 49100 blocked; verify with ss -tlnp | grep 49100.
Client connects, viewport stays black	Stream is up but no scene loaded yet. Either trigger a handler or load a stage manually.
Request returns no reply	serve.py not running, handler not registered, or wire-name mismatch (_request vs :request).
Handler returns unknown scene_id 'X'	Bundled-example case: client scene_id not in the active registry.
Kit log: omni.kit.livestream.messaging: extension not found	Make sure the streaming .kit lists it explicitly under [dependencies].
Multiple clients see the same scene	Expected — LivestreamMessaging is per-app, not per-session.
Camera frames an un-tessellated mesh	Add a 30-60 frame app.next_update_async() settle loop before calling frame_prims.
Python traceback never reaches client	Wrap handler body in try/except and return {"ok": False, "error": ...}. The OmniverseAPI framework does this for you.
Object of type coroutine is not JSON serializable	You're using LivestreamMessaging.observe_and_dispatch directly with an async def handler. The decorator does NOT await coroutines. Use OmniverseAPI's @api.request decorator (handles both sync and async), or split the work into a sync receiver + manual dispatch_event from the async coroutine.

Cross-references

• cae-core/SKILL.md — preflight, Z-up, launch flags.
• cae-core/references/kit-cae-api.md — import_to_stage, execute_command signatures (used by handlers that touch the stage).
• cae-core/references/formats.md — per-format dataset prim paths.
• cae-visualization/SKILL.md — choosing visualization commands inside handlers.
• cae-capture/SKILL.md — for offline rendering, not live viewing.
• references/streaming-protocol.md — the wire spec your handlers must follow.
• references/messaging-api.md — omni.kit.livestream.messaging Python API (the lower layer OmniverseAPI wraps).
• scripts/omniverse_api.py — the @api.request / @api.signal framework.
• scripts/serve.py — runnable reference handler set.