rust-ai-ml-inference-guide

name	rust-ai-ml-inference-guide
description	Use when building Rust AI/ML inference services, model serving infrastructure, or embedded ML pipelines. Covers model loading (GGUF/ONNX/SafeTensors), quantization-aware inference, GPU memory management (CUDA/Metal), tokenizer batching, and embedding vector search. Vertical deepening of rust-architecture-guide for AI inference systems.
metadata	{"version":"1.0.0","philosophy":"Hardware Synergy, Quantization Physics, Batch Efficiency, Jeet Kune Do","domain":"AI/ML inference & model serving","relationship":"vertical-deepening-of:[rust-architecture-guide, rust-systems-cloud-infra-guide]","default_edition":"2024","supported_editions":["2021","2024"],"aligned_with":["candle ML framework","burn deep learning","ort ONNX runtime","llama.cpp GGUF","tokenizers library"]}

Rust AI/ML Inference Guide V1.0.0

Vertical deepening of rust-architecture-guide and rust-systems-cloud-infra-guide for model inference, serving, and embedded ML. Assumes GPU acceleration, quantized models, and high-throughput batching.

Core Philosophy

Principle	Description
Hardware Synergy	Matrix multiplication maps to tensor cores. KV-cache fits in HBM. Attention patterns exploit L2.
Quantization Physics	Q4_0/Q8_0/K-quant map model weights to hardware-friendly int types. 4-bit = 4x less memory bandwidth.
Batch Efficiency	Continuous batching maximizes GPU utilization. Padding waste is minimized via dynamic batching.
Jeet Kune Do	One-pass token generation. KV-cache reuse. Speculative decoding for latency hiding.

---

Action 1: Model Loading & Format Handling

Models come in multiple formats. Rust inference must handle all of them.

• GGUF (llama.cpp format): Quantized weights, tokenizer embedded, metadata. candle/llama-cpp-rs for loading.
• SafeTensors: Safe serialization format. Random access without unsafe deserialization.
• ONNX: Open Neural Network Exchange. ort crate for ONNX Runtime bindings. Cross-framework.
• Red Line: Prohibit loading model weights from untrusted pickle/.pt formats. SafeTensors is a safe format but still requires provenance verification + checksum validation before loading.

→ references/01-model-loading.md

---

Action 2: Quantization-Aware Inference

Quantization reduces model size and accelerates inference without significant quality loss.

• GGUF Quant Types: Q4_0 (4-bit), Q4_K_M (4-bit with K-quant), Q8_0 (8-bit), F16 (half precision)
• Memory Bandwidth Reduction: 4-bit quantization = ~4x less memory bandwidth for weight loading
• candle-quantized: Block-wise quantization. Min/max per block for dequantization.
• Red Line: Quantization introduces quality loss. Must evaluate perplexity/BLEU before deploying.

→ references/02-quantization.md

---

Action 3: GPU Memory Management

GPU VRAM is the bottleneck. Every MB matters.

• KV-Cache: Key-Value cache grows with context length. Pre-allocate based on max context.
• Layer Offloading: Offload some layers to GPU, keep others on CPU (n_gpu_layers in llama.cpp)
• Memory Pool: Pre-allocate large buffers. Avoid per-request allocation/deallocation on GPU.
• Red Line: OOM on GPU = inference failure. Must monitor VRAM usage and apply backpressure.

→ references/03-gpu-memory.md

---

Action 4: Tokenizer & Text Processing

Tokenization is the bridge between raw text and model input.

• tokenizers crate (HuggingFace): BPE/WordPiece/Unigram tokenizers. encode(), decode().
• Special Tokens: BOS, EOS, PAD, UNK, chat template tokens. Must match model training config.
• Batched Encoding: encode_batch() for parallel tokenization. Padding to max length with attention mask.
• Red Line: Mismatched tokenizer = garbage output. Must verify tokenizer.json matches model.

→ references/04-tokenizer.md

---

Action 5: Batching & Continuous Batching

Serving multiple requests concurrently maximizes GPU utilization.

• Static Batching: Pad all requests to same length. Wasteful for variable-length inputs.
• Continuous Batching (vLLM-style): Requests enter/leave batch dynamically. No padding waste.
• Prefill + Decode Splitting: Prefill (process prompt) and decode (generate tokens) as separate GPU operations
• Red Line: Batch requests for throughput serving. For latency-sensitive interactive serving, batch=1 with a continuous batching scheduler is acceptable. Unbatched single-request inference on throughput benchmarks is misleading.

→ references/05-batching.md

---

Action 6: Embedding & Vector Search

Embedding models convert text/images to vectors. Vector search finds nearest neighbors.

• Embedding Models: candle-transformers for BERT/sentence-transformers. Mean pooling for sentence embeddings.
• Vector Database: qdrant/pgvector for storage. HNSW index for approximate nearest neighbor (ANN).
• Cosine Similarity: Normalize vectors to unit length → dot product = cosine similarity
• Red Line: Choose normalization per index type: L2-normalize for cosine/dot-product indexes; do NOT normalize for Euclidean distance indexes. Document the metric per index.

→ references/06-embeddings.md

---

Action 7: Inference Serving & API Design

Model serving requires HTTP/gRPC APIs with streaming support.

• HTTP Streaming: SSE (Server-Sent Events) for token-by-token streaming. axum + tokio-stream.
• gRPC: Protobuf service definition. Bidirectional streaming for interactive use cases.
• Load Balancing: Multiple model replicas. Consistent hashing for KV-cache sticky sessions.
• Red Line: Timeout per request must be enforced. No single request can block the serving thread.

→ references/07-serving-api.md

---

Action 8: Evaluation & Benchmarking

Measure quality and performance before deploying.

• Perplexity: Measure language model quality. Lower = better. Compare before/after quantization.
• Throughput: Tokens/second. Measure at batch sizes 1, 4, 8, 16. GPU vs CPU.
• TTFT (Time to First Token): Latency from request to first token. Critical for UX.
• Red Line: Quantized model must be evaluated against the FP16 baseline. Perplexity degradation limits are task-specific: ≤2% for safety-critical, ≤5% for most production use, ≤8% for throughput-first pipelines where human eval confirms acceptable quality.

→ references/08-evaluation.md

---

Prohibitions Quick List

Category	Prohibited	Mandatory
Model Loading	Untrusted weights without checksum	GGUF checksum / SafeTensors validation
Quantization	Unmeasured quality degradation	Perplexity/BLEU benchmark before deploy
GPU Memory	Unbounded KV-cache growth	Pre-allocation + backpressure on OOM
Tokenizer	Mismatched tokenizer config	Verify tokenizer.json matches model
Batching	Single-request inference	Continuous batching for throughput
Embeddings	Unnormalized vectors for cosine/dot indexes	L2-normalize before storing/searching (per index metric type)
Serving	Unbounded per-request timeout	Timeout + cancellation
Streaming	Blocking token generation	SSE/gRPC streaming protocol
Evaluation	Unmeasured model quality	Perplexity + TTFT + throughput benchmarks

---

Document Relationship Map

flowchart TD
    subgraph Foundation["rust-architecture-guide + cloud-infra-guide"]
        Perf[25-performance-tuning.md]
        IOModel[01-io-model.md]
        Memory[11-memory-advanced.md]
    end

    subgraph ML["rust-ai-ml-inference-guide"]
        Loading[01-model-loading.md]
        Quant[02-quantization.md]
        GPUMem[03-gpu-memory.md]
        Tokenizer[04-tokenizer.md]
        Batching[05-batching.md]
        Embeddings[06-embeddings.md]
        Serving[07-serving-api.md]
        Eval[08-evaluation.md]
    end

    Perf -->|SIMD/memory| Quant
    IOModel -->|async I/O| Serving
    Memory -->|arena/allocator| GPUMem
    Loading --> Quant
    Loading --> Tokenizer
    Quant --> GPUMem
    Tokenizer --> Batching
    Batching --> Serving
    Batching --> Eval
    Embeddings --> Serving

    style Loading fill:#d9534f,stroke:#c9302c,color:#fff
    style Quant fill:#5cb85c,stroke:#4cae4c,color:#fff
    style Batching fill:#f0ad4e,stroke:#ec971f,color:#000

---

Reference Files

File	Topic	Key Directive
01-model-loading.md	Model Loading & Formats	GGUF, SafeTensors, ONNX, checksum verification
02-quantization.md	Quantization-Aware Inference	Q4_0/Q8_0/K-quant, perplexity evaluation requirement
03-gpu-memory.md	GPU Memory Management	KV-cache, layer offloading, memory pool, OOM backpressure
04-tokenizer.md	Tokenizer & Text Processing	BPE/WordPiece, special tokens, batched encoding
05-batching.md	Batching & Continuous Batching	Static vs continuous, prefill/decode splitting
06-embeddings.md	Embedding & Vector Search	BERT embeddings, cosine similarity, HNSW, normalization
07-serving-api.md	Inference Serving & API	SSE streaming, gRPC, load balancing, timeouts
08-evaluation.md	Evaluation & Benchmarking	Perplexity, TTFT, throughput, quantization quality

---

Changelog

V1.0.0

• Initial framework: GGUF/SafeTensors/ONNX model loading, quantization (Q4/Q8/K-quant)
• GPU memory management: KV-cache, layer offloading, OOM backpressure
• HuggingFace tokenizers, continuous batching, prefill/decode splitting
• Embedding models, vector search, cosine similarity normalization
• SSE/gRPC serving API, perplexity/TTFT/throughput evaluation
• Aligned with rust-architecture-guide V9.1.0 and rust-systems-cloud-infra-guide V6.1.0