Hardware acceleration — read this before deploying
Cross-platform ML deployment has sharp edges:
macOS: CoreML EP (auto-selected by ort)
Linux: CUDA EP requires matching ONNX Runtime + driver version
Jetson (aarch64 Linux): mainstream prebuilt ort crates do NOT work — you MUST
recompile ONNX Runtime from source on the Jetson. See HARDWARE_ACCELERATION.zh.md.
Windows: CPU EP works out of the box; CUDA needs manual setup
Current SDK uses ort = "2.0.0-rc.10" which requires ONNX Runtime 1.22.x
For extensions that react to device updates (e.g. running inference on a camera image
whenever it changes):
#[async_trait]
impl Extension for YoloDeviceInference {
fn event_subscriptions(&self) -> &[&str] {
// Declare which event type names you want to receive.
// Empty (the default) means the dispatcher silently drops everything.
&["DeviceDataUpdated"]
}
// SYNC! Use parking_lot::RwLock, not tokio::Mutex.
fn handle_event(&self, event_type: &str, payload: &serde_json::Value) -> Result<()> {
// The dispatcher wraps events in {event_type, payload: {...}, timestamp}.
let inner = payload.get("payload").unwrap_or(payload);
match event_type {
"DeviceDataUpdated" => {
let device_id = inner.get("device_id").and_then(|v| v.as_str())
.ok_or_else(|| ExtensionError::InvalidArguments("missing device_id".into()))?;
// ... kick off inference on a tokio task; sync handoff to internal channel
}
_ => {}
}
Ok(())
}
}
Gotchas (all caused real bugs)
Forgot to override event_subscriptions() → default &[] → dispatcher filters out
every event silently. Always override when you expect events.
Used tokio::Mutex for shared state → deadlock because handle_event is sync.
Use parking_lot::RwLock / parking_lot::Mutex instead.
Read payload.get("session_id") directly → wrong level; the actual delivered shape
is {event_type, payload: {session_id, ...}, timestamp}. Always unwrap with
payload.get("payload").unwrap_or(payload) first.
String casing on event types — agent stream terminators emit "type": "end"
(lowercase). Match both "end" and "End".
Never use Tailwind — extension bundles don't ship Tailwind. Use NeoMind CSS
variables for all colors: var(--foreground), var(--card), var(--border), etc.
Never hardcode colors (#fff, rgb(...)) — they break dark mode.
Primary button text must use var(--{prefix}-on-primary), not
var(--primary-foreground) or #fff. See design guide §5.1.
UMD format, React/ReactDOM external (provided by host).
Component type in frontend.json must be unique — duplicate types collide in the
UI. The build script auto-generates types as {extension-name-without-v2}-card.
Every component: forwardRef, handle loading/error/empty states, scoped CSS with
extension-prefixed class names (.weather-, .yolo-, .deep-stream-).
Bridge extensions import external systems (Modbus / LoRaWAN / Home Assistant / OPC UA /
ONVIF / BACnet / custom REST) into NeoMind's device model. Reference impls in
extensions/{modbus,lorawan,homeassistant,opcua,onvif,bacnet}-bridge/.
Pattern summary
Connect command (connect, add_device) — user provides address + credentials,
extension establishes a client and starts a background polling / listening task.
Auto-discovery — fetch device list from the external system; for each discovered
device, register a NeoMind device via device_register.
Background worker — dedicated thread reads data continuously and updates an
in-memory cache (Arc<parking_lot::Mutex<DeviceState>> or Arc<RwLock<...>>).
produce_metrics() fan-out — periodically writes per-device metrics via
device_metrics_write capability.
Resilience — exponential backoff (typically 1s → 60s), separate counters for
transient vs auth failures (HA bridge gives up after 5 auth failures).
Capabilities used
Bridges invoke these dynamically via CapabilityContext::default():
Capability
When
device_template_register
Once on first device — registers the metric/command schema
device_register
For each discovered device
device_metrics_write
On every poll cycle, for each device metric
device_unregister
On remove_device / disconnect
Choosing your thread model
Client type
Use
Spawn pattern
Example
Sync (modbus, ureq)
std::thread::spawn
`Builder::new().name(...).spawn(move
Async (MQTT, WS)
tokio::spawn on the host runtime
Handle::try_current()?.spawn(async move {...})
lorawan-bridge, HA bridge WS loop
Mixed (REST + WS)
Sync REST from anywhere + async WS on host runtime
HA bridge
Never call tokio::runtime::Runtime::new() inside an extension — it collides with the
runner's runtime. To run async work, grab the host's Handle::try_current() and spawn there.
Skeleton (modbus-style sync polling)
use neomind_extension_sdk::host::CapabilityContext;
use serde_json::{json, Value};
pub struct ModbusBridge {
devices: Arc<parking_lot::RwLock<HashMap<String, DeviceState>>>,
template_registered: AtomicBool,
}
impl ModbusBridge {
fn register_template(&self) {
// Guard with atomic flag — idempotent
if self.template_registered.swap(true, Ordering::SeqCst) {
return;
}
let ctx = CapabilityContext::default();
let tpl = json!({
"device_type": "modbus_device",
"name": "Modbus Device",
"metrics": [
{ "name": "connected", "display_name": "Connected", "data_type": "String" },
{ "name": "poll_errors", "display_name": "Poll Errors", "data_type": "Integer" },
],
"commands": [],
});
let r = ctx.invoke_capability("device_template_register", &tpl);
if !r.get("success").and_then(|v| v.as_bool()).unwrap_or(false) {
self.template_registered.store(false, Ordering::SeqCst); // retry later
}
}
fn register_device(&self, device_id: &str, name: &str) {
let ctx = CapabilityContext::default();
let _ = ctx.invoke_capability("device_register", &json!({
"device_id": device_id,
"name": name,
"device_type": "modbus_device",
}));
}
}
// Background poller — std::thread, persistent connection, poll-level reconnect
fn polling_loop(
config: DeviceConfig,
state: Arc<parking_lot::Mutex<DeviceState>>,
running: Arc<AtomicBool>,
) {
let mut ctx: Option<sync::Context> = None; // persistent connection
while running.load(Ordering::SeqCst) {
let interval = state.lock().config.poll_interval_ms;
let start = std::time::Instant::now();
if ctx.is_none() {
ctx = connect_sync(&config).ok();
}
if let Some(ref mut c) = ctx {
match c.read_holding_registers(addr, count) {
Ok(data) => {
let mut s = state.lock();
s.register_values = parse(data);
s.poll_errors = 0;
s.connected = true;
}
Err(_) => {
ctx = None; // force reconnect next cycle
state.lock().poll_errors += 1;
}
}
}
let elapsed = start.elapsed().as_millis() as u64;
let sleep_ms = interval.saturating_sub(elapsed);
std::thread::sleep(Duration::from_millis(sleep_ms));
}
}
// Sync produce_metrics fans out per-device writes via capability
fn produce_metrics(&self) -> Result<Vec<ExtensionMetricValue>> {
let ctx = CapabilityContext::default();
let now = chrono::Utc::now().timestamp_millis();
let devices = self.devices.read();
for (id, st) in devices.iter() {
let _ = ctx.invoke_capability("device_metrics_write", &json!({
"device_id": id, "metric": "connected",
"value": if st.connected { "true" } else { "false" },
"timestamp": now,
}));
for rv in &st.register_values {
let _ = ctx.invoke_capability("device_metrics_write", &json!({
"device_id": id, "metric": rv.name,
"value": rv.value, "timestamp": now,
}));
}
}
Ok(vec![]) // extension-level metrics optional
}
Reconnect strategies
HA bridge — exponential backoff 1s → 60s, separate counter for AuthFailed (stops
after 5), REST resync on every WS reconnect to catch missed state_changed events.
LoRaWAN bridge — rumqttc auto-reconnects at MQTT level; extension-level 500ms → 30s
backoff for stream errors; re-subscribes on ConnAck.
Modbus bridge — poll-level retry; persistent TCP connection; full reconnect on
read error.
When you need Python libraries (CosyVoice, edge-tts, sherpa-onnx ASR, PaddleOCR, etc.),
keep Rust thin and put the AI logic in a separate Python service. Reference impls:
voice-assistant, voice-edge-tts, cosyvoice-3, moss-tts-nano, sensevoice-asr.
Use when you need continuous streaming (PCM audio in/out, ASR → LLM → TTS pipeline).
See extensions/voice-assistant/src/lib.rs — run_session_pump is the canonical WS
loop with tokio::select! over browser PCM input and Python event output.
Beyond plain commands, extensions can invoke host capabilities via
CapabilityContext::default().invoke_capability(name, &json). The SDK also provides
typed helpers in neomind_extension_sdk::capabilities::{device, chat}.
ChatStream — streaming LLM without managing API tokens
The biggest capability addition. Two usage patterns:
Phase 1: one-shot (simplest)
use neomind_extension_sdk::capabilities::chat;
use neomind_extension_sdk::host::CapabilityContext;
let ctx = CapabilityContext::default();
let result = chat::invoke(&ctx, "Hello, what's the weather?", None).await?;
let sid = result["session_id"].as_str().unwrap();
// LLM tokens arrive as AgentStreamChunk events (see Step 7 / Step 12)
Phase 2: persistent session (multi-turn)
let ctx = CapabilityContext::default();
// 1. Open (or reuse) a session
let open = chat::open_session(&ctx, None).await?; // {session_id, created}
let sid = open["session_id"].as_str().unwrap().to_string();
// 2. Send a turn — returns immediately with turn_id
let turn = chat::send_message(&ctx, &sid, "What about tomorrow?").await?;
let turn_id = turn["turn_id"].as_str().unwrap().to_string();
// Tokens stream in via AgentStreamChunk events tagged with this turn_id
// 3. (optional) Cancel an in-flight turn without closing the session
chat::cancel_turn(&ctx, &sid, Some(&turn_id)).await?;
// 4. Close when truly done
chat::close_session(&ctx, &sid).await?;
Events you must subscribe to (override event_subscriptions()):
AgentStreamChunk — one token / chunk. Payload: {session_id, chunk: {type, content}, timestamp}.
chunk.type can be "Content", "reasoning", or "end" (lowercase!) — "end" is not
authoritative on its own because reasoning models emit intermediate ends.
AgentStreamEnd — authoritative terminator. Payload: {session_id, reason, error, timestamp}.
Use this to clean up session state.
Other useful capabilities
Capability
Helper
Purpose
device_metrics_read
device::get_metrics
Read another device's current metrics
device_metrics_write
device::write_virtual_metric
Write virtual metrics for your own devices
device_control
device::send_command
Send control commands to devices
telemetry_history
device::query_telemetry_last_24h
Historical metric data
metrics_aggregate
device::aggregate_avg_24h
Aggregated metrics (avg / min / max)
storage_query
—
Query platform storage
extension_call
—
Call commands on other extensions
event_publish
—
Publish events to other extensions
agent_trigger
—
Trigger NeoMind AI agents
rule_trigger
—
Trigger automation rules
See reference/sdk-api.md → "CapabilityContext" for the full list and exact return shapes.
Step 12: Python↔Rust ChatStream Bridge (voice-assistant full loop)
Real-world pattern from voice-assistant: Python side wants streaming LLM access without
holding tokens. The Rust side brokers between Python WS frames and the platform
ChatStream capability.
Store per-session state in Arc<parking_lot::RwLock<HashMap<String, mpsc::Sender<...>>>>
keyed by session_id. Do not remove the entry on every turn — only on WS teardown.
Otherwise you force a redundant chat_session_open round-trip per turn.
// ❌ Bad — blocks the executor
async fn execute_command(&self, ...) -> Result<Value> {
std::thread::sleep(Duration::from_secs(5));
Ok(json!({}))
}
// ✅ Good
async fn execute_command(&self, ...) -> Result<Value> {
tokio::time::sleep(Duration::from_secs(5)).await;
Ok(json!({}))
}
// ✅ Also good — sync work wrapped in spawn_blocking
async fn execute_command(&self, ...) -> Result<Value> {
tokio::task::spawn_blocking(|| { /* sync work */ }).await?
}
HTTP clients — always ureq
Async HTTP clients (reqwest, hyper) create their own Tokio runtime, which panics
inside a cdylib. Use ureq (sync). If you must call it from an async command, wrap in
tokio::task::spawn_blocking. See reference/sdk-api.md for the canonical pattern.
Thread safety
Arc<parking_lot::Mutex<T>> for state shared between sync (handle_event) and async
(execute_command) code paths.
Arc<tokio::sync::Mutex<T>> only when the lock is held across .await points AND never
touched from handle_event.
Arc<AtomicI64> / AtomicBool for simple counters / flags.
Marketplace Release Checklist (CRITICAL)
1. Component type MUST be unique across all extensions
# Verify in every .nep you're about to ship:
unzip -p dist/your-extension-2.3.1-darwin_aarch64.nep manifest.json \
| jq '.frontend.components[].type'
Duplicate types cause only one of the conflicting extensions to appear in the UI. The
build script auto-generates the type as {extension-name-without-v2}-card.
2. frontend.components in metadata.json MUST be a string array
