ai-assisted-coding

name	ai-assisted-coding
description	Build or maintain AI-assisted coding features in Rust using Onde Inference. Use when working on ChatEngine integration, model loading, streaming inference, history management, sampling config, or local coding-agent architecture.

Skill: AI-Assisted Coding Agents — Onde Inference Integration

Overview

Building a local AI coding agent in Rust using Onde Inference as the LLM backend. Onde wraps mistral.rs with a clean API for model loading, history management, and streaming inference across macOS (Metal), iOS, Android, Linux, and Windows.

Crate: onde = "1.1.2" (published on crates.io; siGit pins it in Cargo.toml) Repo: https://github.com/ondeinference/onde Docs: https://ondeinference.com

---

Onde ChatEngine API

Construction and lifecycle

use onde::inference::{ChatEngine, GgufModelConfig, SamplingConfig};

let engine = ChatEngine::new();        // starts unloaded
engine.is_loaded().await               // -> bool
engine.unload_model().await            // -> ()

Loading a model

// Platform-aware default (Qwen 2.5 3B on macOS, 1.5B on iOS/tvOS/Android)
let config = GgufModelConfig::platform_default();

// Load — blocks until model is in memory and on GPU
engine
    .load_gguf_model(
        config,
        Some("You are a helpful assistant.".to_string()),  // system prompt
        None,  // sampling config (uses SamplingConfig::default() internally)
    )
    .await?;

// AlreadyLoaded error if called twice — check first:
if !engine.is_loaded().await {
    engine.load_gguf_model(...).await?;
}

Model sizes (macOS/Windows/Linux default — Qwen 2.5 3B Q4_K_M): ~1.93 GB
Model sizes (iOS/tvOS/Android default — Qwen 2.5 1.5B Q4_K_M): ~941 MB
First run downloads from HuggingFace Hub into ~/.cache/huggingface/.

Blocking (non-streaming) inference

let result = engine.send_message("What is Rust's ownership model?").await?;
// result: InferenceResult
println!("{}", result.text);
println!("took {}", result.duration_display);  // e.g. "3.2s"

send_message appends both the user message and assistant reply to conversation history automatically.

Streaming inference

let mut rx: tokio::sync::mpsc::Receiver<StreamChunk> =
    engine.stream_message("Tell me a story.").await?;

while let Some(chunk) = rx.recv().await {
    if !chunk.delta.is_empty() {
        print!("{}", chunk.delta);   // partial token text
    }
    if chunk.done {
        // chunk.finish_reason: Option<String> — e.g. "stop", "length"
        break;
    }
}

StreamChunk fields:

• delta: String — the new token(s) in this chunk
• done: bool — true on the last chunk
• finish_reason: Option<String> — present on final chunk only

History is updated automatically after the stream completes.

One-shot generation (no history side-effects)

use onde::inference::ChatMessage;

let result = engine.generate(
    vec![ChatMessage::user("Expand: a cat in space")],
    Some(SamplingConfig::deterministic()),
).await?;
println!("{}", result.text);
// Does NOT modify conversation history

History management

let history: Vec<ChatMessage> = engine.history().await;
let removed: usize = engine.clear_history().await;  // returns count cleared
engine.push_history(ChatMessage::user("context")).await;
engine.set_system_prompt("new system prompt").await;
engine.clear_system_prompt().await;

Engine status

let info: EngineInfo = engine.info().await;
// info.status: EngineStatus (Unloaded | Loading | Ready | Generating | Error)
// info.model_name: Option<String>
// info.approx_memory: Option<String>  e.g. "~1.93 GB"
// info.history_length: u64

---

InferenceError variants

match err {
    InferenceError::NoModelLoaded       => { /* load model first */ }
    InferenceError::AlreadyLoaded { model_name } => { /* already loaded */ }
    InferenceError::ModelBuild { reason } => { /* load failure */ }
    InferenceError::Inference { reason }  => { /* runtime inference error */ }
    InferenceError::Cancelled            => { /* was cancelled */ }
    InferenceError::Other { reason }     => { /* unexpected */ }
}

Map to ACP errors:

.map_err(|e| agent_client_protocol::Error::new(-32603, e.to_string()))?

---

SamplingConfig presets

Preset	temp	top_p	max_tokens	Use case
SamplingConfig::default()	0.7	0.95	512	General chat
SamplingConfig::deterministic()	0.0	—	512	Code / reproducible
SamplingConfig::mobile()	0.7	0.95	128	Memory-constrained
SamplingConfig::coding()	0.0	—	512	Code generation
SamplingConfig::coding_mobile()	0.0	—	128	Code on mobile

---

GgufModelConfig constructors

GgufModelConfig::platform_default()    // auto-selects based on target_os
GgufModelConfig::qwen25_1_5b()         // force 1.5B
GgufModelConfig::qwen25_3b()           // force 3B
GgufModelConfig::qwen25_coder_1_5b()   // coder variant 1.5B
GgufModelConfig::qwen25_coder_3b()     // coder variant 3B
GgufModelConfig::qwen25_coder_7b()     // coder variant 7B (tool calling)
GgufModelConfig::qwen3_1_7b()          // Qwen 3 1.7B (tool calling)
GgufModelConfig::qwen3_4b()            // Qwen 3 4B (tool calling)
GgufModelConfig::qwen3_8b()            // Qwen 3 8B (tool calling)
GgufModelConfig::qwen3_14b()           // Qwen 3 14B (tool calling)

Only the Qwen 3 family and Qwen 2.5 Coder 7B support tool calling — see the tool-calling skill. The on-device default is the saved selection, falling back to platform_default() (Qwen 2.5 3B on macOS).

---

Adding onde as a Rust library dependency

# In your crate's Cargo.toml — onde is published on crates.io
onde = "1.1.2"
# For local SDK development against a checkout, swap to a path dep:
# onde = { path = "../onde" }

Important: onde declares crate-type = ["lib", "cdylib", "staticlib"]. When used as a Rust library dep, only the lib target is compiled. The cdylib/staticlib targets (used for Swift/Kotlin FFI) are not built. The uniffi::setup_scaffolding!() macro generates #[no_mangle] extern "C" symbols but these are harmless in a binary context.

The [patch.crates-io] in onde's Cargo.toml does NOT propagate to dependents unless they are in the same workspace. The sysctl patch is only needed for watchOS; macOS/iOS/Linux work without it.

GPU feature selection is automatic via target_os cfg flags in onde's Cargo.toml — you get Metal on macOS/iOS without any extra features in your crate.

---

Patterns for coding agents

Single-engine, multi-session via history reset

For a simple MVP where one session is active at a time:

struct MyAgent {
    engine: Arc<ChatEngine>,
    active_session: Arc<Mutex<Option<SessionId>>>,
}

// new_session handler:
if self.engine.is_loaded().await {
    self.engine.clear_history().await;   // reuse model, fresh conversation
} else {
    self.engine
        .load_gguf_model(GgufModelConfig::platform_default(), Some(SYSTEM_PROMPT.into()), None)
        .await?;
}

Why: Loading the model is expensive (seconds + GB of RAM). Reloading for each session would make the agent feel broken. clear_history() resets context in microseconds.

Per-session engines (multiple concurrent sessions)

When you need truly isolated parallel sessions:

use std::collections::HashMap;

struct MultiSessionAgent {
    sessions: Arc<Mutex<HashMap<String, Arc<ChatEngine>>>>,
}

// new_session: create and load a new engine per session
// prompt: look up session engine, call send_message or stream_message
// CAVEAT: each engine holds a separate model copy in GPU memory — expensive!

Better approach for shared GPU memory: use engine.generate() (no history side-effects) with an explicitly managed message vec per session.

System prompt design for coding agents

const SYSTEM_PROMPT: &str = "\
You are <AgentName>, an expert AI coding agent integrated into your editor \
via the Agent Client Protocol. You specialize in:

- Code analysis, writing, and refactoring
- Bug hunting and debugging
- Git workflows and commit messages
- Software architecture and design patterns
- Code review and best practices

Be concise, precise, and practical. Write clean, idiomatic code with brief \
explanations. Identify root causes when debugging. Prefer correctness over brevity.";

Key principles:

• State the agent's role and name clearly (models respond better to named personas)
• List specializations explicitly (influences which parts of training are activated)
• Set tone expectations: "concise", "practical", "idiomatic"
• Avoid verbose instruction lists — they cost tokens on every turn

Streaming tokens to ACP (connecting onde → ACP)

// In the prompt handler — cx: &ConnectionTo<Client> is passed in by the builder
// (agent-client-protocol 0.13). No mpsc forwarder; send through cx directly.
let mut rx = self.engine.stream_message(user_text).await
    .map_err(|e| Error::new(-32603, e.to_string()))?;

while let Some(chunk) = rx.recv().await {
    if !chunk.delta.is_empty() {
        cx.send_notification(SessionNotification::new(
            session_id.clone(),
            SessionUpdate::AgentMessageChunk(
                ContentChunk::new(ContentBlock::from(chunk.delta)),
            ),
        )).ok();  // .ok() — ignore if the client is gone
    }
    if chunk.done { break; }
}

Ok(PromptResponse::new(StopReason::EndTurn))

The PromptResponse is returned AFTER the stream finishes. The client receives streaming tokens via session/update notifications while blocking on the session/prompt response. See the agent-client-protocol skill for the cx (ConnectionTo<Client>) model that replaced the old mpsc-channel forwarder.

Note: siGit's actual handle_prompt does not stream token-by-token — it runs a tool-calling loop through an InferenceBackend and sends the final text in one AgentMessageChunk. The streaming pattern above still applies if you want incremental output. See the tool-calling skill for the backend loop.

---

Extracting text from ACP PromptRequest

ACP prompts can contain text, images, resource links, etc. For a text-only coding agent:

let user_text: String = args.prompt.iter()
    .filter_map(|block| match block {
        ContentBlock::Text(t) => Some(t.text.as_str()),
        // Skip images, resource links, embedded resources for now
        _ => None,
    })
    .collect::<Vec<_>>()
    .join("\n");

For resource context (e.g. open files provided by Zed) — note the variant is TextResourceContents, not Text:

ContentBlock::Resource(r) => match &r.resource {
    EmbeddedResourceResource::TextResourceContents(t) => Some(t.text.as_str()),
    EmbeddedResourceResource::BlobResourceContents(_) => None,
    _ => None,
},

siGit also handles ContentBlock::ResourceLink (a file:// reference it reads from disk, including #L<start>:<end> line-range fragments). See the tool-calling skill.

---

ChatEngine threading model

• Internally uses Arc<tokio::sync::Mutex<Option<LoadedModel>>> — Send + Sync.
• Safe to wrap in Arc<ChatEngine> and share across tasks.
• stream_message() spawns a tokio::spawn background task internally — the mistralrs model must be Send, which it is on all supported platforms.
• block_in_place trap: load_gguf_model calls tokio::task::block_in_place internally, which panics unless it's on a multi-threaded runtime worker. Run model loads on a dedicated std::thread with its own tokio::runtime::Runtime and signal back via AtomicBool/oneshot. siGit does exactly this in both ACP and TUI modes — see the agent-client-protocol and tool-calling skills.

---

First-run model download

On first use, onde downloads the GGUF model from HuggingFace Hub:

• Requires internet connectivity
• Cached at ~/.cache/huggingface/ (or HF_HUB_CACHE env var)
• HF_TOKEN env var needed for gated models (public Qwen models don't need it)
• Subsequent runs load from disk cache — fast

For sandboxed environments (iOS, tvOS, Android):

• Set HF_HOME and HF_HUB_CACHE to a path inside the app container
• Do this BEFORE calling any ChatEngine method
• See onde/docs/swift-package.md for setupInferenceEnvironment() pattern

---

Common mistakes

1. Calling load_gguf_model twice without checking is_loaded() first → InferenceError::AlreadyLoaded. Always guard with is_loaded().await.

2. Blocking on the stream after the channel is closed → the stream naturally ends when the done flag is true. Don't recv() after done.

3. Losing StreamChunk deltas when delta is empty (whitespace tokens) → always check !chunk.delta.is_empty() before sending to avoid empty notifications that waste bandwidth.

4. Sharing one ChatEngine across parallel prompts without coordination → the internal Mutex serializes inference, so concurrent prompts queue up. Design for sequential access per engine instance.

5. Using SamplingConfig::default() for code generation → prefer SamplingConfig::coding() (deterministic, temp=0) for more reliable code output.

6. Forgetting that generate() doesn't update history — use it for one-shot enhancements (prompt expansion, code review) that shouldn't pollute the main conversation. Use send_message() / stream_message() for the primary turn loop.