Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.
Quelldateien prüfen
Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
Hardware acceleration — read this before deploying
Cross-platform ML deployment has sharp edges:
macOS: CoreML EP (auto-selected by ort)
Linux: CUDA EP requires matching ONNX Runtime + driver version
Jetson (aarch64 Linux): mainstream prebuilt ort crates do NOT work — you MUST
recompile ONNX Runtime from source on the Jetson. See HARDWARE_ACCELERATION.zh.md.
Windows: CPU EP works out of the box; CUDA needs manual setup
Current SDK uses ort = "2.0.0-rc.10" which requires ONNX Runtime 1.22.x
For extensions that react to device updates (e.g. running inference on a camera image
whenever it changes):
#[async_trait]
impl Extension for YoloDeviceInference {
fn event_subscriptions(&self) -> &[&str] {
// Declare which event type names you want to receive.
// Empty (the default) means the dispatcher silently drops everything.
&["DeviceDataUpdated"]
}
// SYNC! Use parking_lot::RwLock, not tokio::Mutex.
fn handle_event(&self, event_type: &str, payload: &serde_json::Value) -> Result<()> {
// The dispatcher wraps events in {event_type, payload: {...}, timestamp}.
let inner = payload.get("payload").unwrap_or(payload);
match event_type {
"DeviceDataUpdated" => {
let device_id = inner.get("device_id").and_then(|v| v.as_str())
.ok_or_else(|| ExtensionError::InvalidArguments("missing device_id".into()))?;
// ... kick off inference on a tokio task; sync handoff to internal channel
}
_ => {}
}
Ok(())
}
}
Gotchas (all caused real bugs)
Forgot to override event_subscriptions() → default &[] → dispatcher filters out
every event silently. Always override when you expect events.
Used tokio::Mutex for shared state → deadlock because handle_event is sync.
Use parking_lot::RwLock / parking_lot::Mutex instead.
Read payload.get("session_id") directly → wrong level; the actual delivered shape
is {event_type, payload: {session_id, ...}, timestamp}. Always unwrap with
payload.get("payload").unwrap_or(payload) first.
String casing on event types — agent stream terminators emit "type": "end"
(lowercase). Match both "end" and "End".
Never use Tailwind — extension bundles don't ship Tailwind. Use NeoMind CSS
variables for all colors: var(--foreground), var(--card), var(--border), etc.
Never hardcode colors (#fff, rgb(...)) — they break dark mode.
Primary button text must use var(--{prefix}-on-primary), not
var(--primary-foreground) or #fff. See design guide §5.1.
UMD format, React/ReactDOM external (provided by host).
Component type in frontend.json must be unique — duplicate types collide in the
UI. The build script auto-generates types as {extension-name-without-v2}-card.
Every component: forwardRef, handle loading/error/empty states, scoped CSS with
extension-prefixed class names (.weather-, .yolo-, .deep-stream-).
Bridge extensions import external systems (Modbus / LoRaWAN / Home Assistant / OPC UA /
ONVIF / BACnet / custom REST) into NeoMind's device model. Reference impls in
extensions/{modbus,lorawan,homeassistant,opcua,onvif,bacnet}-bridge/.
Pattern summary
Connect command (connect, add_device) — user provides address + credentials,
extension establishes a client and starts a background polling / listening task.
Auto-discovery — fetch device list from the external system; for each discovered
device, register a NeoMind device via device_register.
Background worker — dedicated thread reads data continuously and updates an
in-memory cache (Arc<parking_lot::Mutex<DeviceState>> or Arc<RwLock<...>>).
produce_metrics() fan-out — periodically writes per-device metrics via
device_metrics_write capability.
Resilience — exponential backoff (typically 1s → 60s), separate counters for
transient vs auth failures (HA bridge gives up after 5 auth failures).
Capabilities used
Bridges invoke these dynamically via CapabilityContext::default():
Capability
When
device_template_register
Once on first device — registers the metric/command schema
device_register
For each discovered device
device_metrics_write
On every poll cycle, for each device metric
device_unregister
On remove_device / disconnect
Choosing your thread model
Client type
Use
Spawn pattern
Example
Sync (modbus, ureq)
std::thread::spawn
`Builder::new().name(...).spawn(move
Async (MQTT, WS)
tokio::spawn on the host runtime
Handle::try_current()?.spawn(async move {...})
lorawan-bridge, HA bridge WS loop
Mixed (REST + WS)
Sync REST from anywhere + async WS on host runtime
HA bridge
Never call tokio::runtime::Runtime::new() inside an extension — it collides with the
runner's runtime. To run async work, grab the host's Handle::try_current() and spawn there.
Skeleton (modbus-style sync polling)
use neomind_extension_sdk::host::CapabilityContext;
use serde_json::{json, Value};
pub struct ModbusBridge {
devices: Arc<parking_lot::RwLock<HashMap<String, DeviceState>>>,
template_registered: AtomicBool,
}
impl ModbusBridge {
fn register_template(&self) {
// Guard with atomic flag — idempotent
if self.template_registered.swap(true, Ordering::SeqCst) {
return;
}
let ctx = CapabilityContext::default();
let tpl = json!({
"device_type": "modbus_device",
"name": "Modbus Device",
"metrics": [
{ "name": "connected", "display_name": "Connected", "data_type": "String" },
{ "name": "poll_errors", "display_name": "Poll Errors", "data_type": "Integer" },
],
"commands": [],
});
let r = ctx.invoke_capability("device_template_register", &tpl);
if !r.get("success").and_then(|v| v.as_bool()).unwrap_or(false) {
self.template_registered.store(false, Ordering::SeqCst); // retry later
}
}
fn register_device(&self, device_id: &str, name: &str) {
let ctx = CapabilityContext::default();
let _ = ctx.invoke_capability("device_register", &json!({
"device_id": device_id,
"name": name,
"device_type": "modbus_device",
}));
}
}
// Background poller — std::thread, persistent connection, poll-level reconnect
fn polling_loop(
config: DeviceConfig,
state: Arc<parking_lot::Mutex<DeviceState>>,
running: Arc<AtomicBool>,
) {
let mut ctx: Option<sync::Context> = None; // persistent connection
while running.load(Ordering::SeqCst) {
let interval = state.lock().config.poll_interval_ms;
let start = std::time::Instant::now();
if ctx.is_none() {
ctx = connect_sync(&config).ok();
}
if let Some(ref mut c) = ctx {
match c.read_holding_registers(addr, count) {
Ok(data) => {
let mut s = state.lock();
s.register_values = parse(data);
s.poll_errors = 0;
s.connected = true;
}
Err(_) => {
ctx = None; // force reconnect next cycle
state.lock().poll_errors += 1;
}
}
}
let elapsed = start.elapsed().as_millis() as u64;
let sleep_ms = interval.saturating_sub(elapsed);
std::thread::sleep(Duration::from_millis(sleep_ms));
}
}
// Sync produce_metrics fans out per-device writes via capability
fn produce_metrics(&self) -> Result<Vec<ExtensionMetricValue>> {
let ctx = CapabilityContext::default();
let now = chrono::Utc::now().timestamp_millis();
let devices = self.devices.read();
for (id, st) in devices.iter() {
let _ = ctx.invoke_capability("device_metrics_write", &json!({
"device_id": id, "metric": "connected",
"value": if st.connected { "true" } else { "false" },
"timestamp": now,
}));
for rv in &st.register_values {
let _ = ctx.invoke_capability("device_metrics_write", &json!({
"device_id": id, "metric": rv.name,
"value": rv.value, "timestamp": now,
}));
}
}
Ok(vec![]) // extension-level metrics optional
}
Reconnect strategies
HA bridge — exponential backoff 1s → 60s, separate counter for AuthFailed (stops
after 5), REST resync on every WS reconnect to catch missed state_changed events.
LoRaWAN bridge — rumqttc auto-reconnects at MQTT level; extension-level 500ms → 30s
backoff for stream errors; re-subscribes on ConnAck.
Modbus bridge — poll-level retry; persistent TCP connection; full reconnect on
read error.
When you need Python libraries (CosyVoice, edge-tts, sherpa-onnx ASR, PaddleOCR, etc.),
keep Rust thin and put the AI logic in a separate Python service. Reference impls:
voice-assistant, voice-edge-tts, cosyvoice-3, moss-tts-nano, sensevoice-asr.
Use when you need continuous streaming (PCM audio in/out, ASR → LLM → TTS pipeline).
See extensions/voice-assistant/src/lib.rs — run_session_pump is the canonical WS
loop with tokio::select! over browser PCM input and Python event output.
Beyond plain commands, extensions can invoke host capabilities via
CapabilityContext::default().invoke_capability(name, &json). The SDK also provides
typed helpers in neomind_extension_sdk::capabilities::{device, chat}.
ChatStream — streaming LLM without managing API tokens
The biggest capability addition. Two usage patterns:
Phase 1: one-shot (simplest)
use neomind_extension_sdk::capabilities::chat;
use neomind_extension_sdk::host::CapabilityContext;
let ctx = CapabilityContext::default();
let result = chat::invoke(&ctx, "Hello, what's the weather?", None).await?;
let sid = result["session_id"].as_str().unwrap();
// LLM tokens arrive as AgentStreamChunk events (see Step 7 / Step 12)
Phase 2: persistent session (multi-turn)
let ctx = CapabilityContext::default();
// 1. Open (or reuse) a session
let open = chat::open_session(&ctx, None).await?; // {session_id, created}
let sid = open["session_id"].as_str().unwrap().to_string();
// 2. Send a turn — returns immediately with turn_id
let turn = chat::send_message(&ctx, &sid, "What about tomorrow?").await?;
let turn_id = turn["turn_id"].as_str().unwrap().to_string();
// Tokens stream in via AgentStreamChunk events tagged with this turn_id
// 3. (optional) Cancel an in-flight turn without closing the session
chat::cancel_turn(&ctx, &sid, Some(&turn_id)).await?;
// 4. Close when truly done
chat::close_session(&ctx, &sid).await?;
Events you must subscribe to (override event_subscriptions()):
AgentStreamChunk — one token / chunk. Payload: {session_id, chunk: {type, content}, timestamp}.
chunk.type can be "Content", "reasoning", or "end" (lowercase!) — "end" is not
authoritative on its own because reasoning models emit intermediate ends.
AgentStreamEnd — authoritative terminator. Payload: {session_id, reason, error, timestamp}.
Use this to clean up session state.
Other useful capabilities
Capability
Helper
Purpose
device_metrics_read
device::get_metrics
Read another device's current metrics
device_metrics_write
device::write_virtual_metric
Write virtual metrics for your own devices
device_control
device::send_command
Send control commands to devices
telemetry_history
device::query_telemetry_last_24h
Historical metric data
metrics_aggregate
device::aggregate_avg_24h
Aggregated metrics (avg / min / max)
storage_query
—
Query platform storage
extension_call
—
Call commands on other extensions
event_publish
—
Publish events to other extensions
agent_trigger
—
Trigger NeoMind AI agents
rule_trigger
—
Trigger automation rules
See reference/sdk-api.md → "CapabilityContext" for the full list and exact return shapes.
Step 12: Python↔Rust ChatStream Bridge (voice-assistant full loop)
Real-world pattern from voice-assistant: Python side wants streaming LLM access without
holding tokens. The Rust side brokers between Python WS frames and the platform
ChatStream capability.
Store per-session state in Arc<parking_lot::RwLock<HashMap<String, mpsc::Sender<...>>>>
keyed by session_id. Do not remove the entry on every turn — only on WS teardown.
Otherwise you force a redundant chat_session_open round-trip per turn.
// ❌ Bad — blocks the executor
async fn execute_command(&self, ...) -> Result<Value> {
std::thread::sleep(Duration::from_secs(5));
Ok(json!({}))
}
// ✅ Good
async fn execute_command(&self, ...) -> Result<Value> {
tokio::time::sleep(Duration::from_secs(5)).await;
Ok(json!({}))
}
// ✅ Also good — sync work wrapped in spawn_blocking
async fn execute_command(&self, ...) -> Result<Value> {
tokio::task::spawn_blocking(|| { /* sync work */ }).await?
}
HTTP clients — always ureq
Async HTTP clients (reqwest, hyper) create their own Tokio runtime, which panics
inside a cdylib. Use ureq (sync). If you must call it from an async command, wrap in
tokio::task::spawn_blocking. See reference/sdk-api.md for the canonical pattern.
Thread safety
Arc<parking_lot::Mutex<T>> for state shared between sync (handle_event) and async
(execute_command) code paths.
Arc<tokio::sync::Mutex<T>> only when the lock is held across .await points AND never
touched from handle_event.
Arc<AtomicI64> / AtomicBool for simple counters / flags.
Marketplace Release Checklist (CRITICAL)
1. Component type MUST be unique across all extensions
# Verify in every .nep you're about to ship:
unzip -p dist/your-extension-2.3.1-darwin_aarch64.nep manifest.json \
| jq '.frontend.components[].type'
Duplicate types cause only one of the conflicting extensions to appear in the UI. The
build script auto-generates the type as {extension-name-without-v2}-card.
2. frontend.components in metadata.json MUST be a string array
