colibri

name	colibri
description	Assist with Colibri: pure-C LLM inference engine for running GLM-5.2 (744B MoE) on consumer machines with ~25 GB RAM. Use when setting up, building, converting models, running inference, configuring expert streaming and caching, optimizing speculative decoding (MTP), GPU integration, and integrating Colibri into production pipelines. Includes build setup, model download & conversion, chat/inference modes, performance tuning, and API integration patterns.
allowed-tools	Read Grep Glob Bash Write Edit
compatibility	Works with Colibri (main branch). Useful for AI engineers, ML researchers, and developers needing efficient LLM inference on consumer hardware or edge devices. Pairs well with model quantization, local inference infrastructure, and LLM application development skills.
metadata	{"tags":"colibri, llm-inference, glm-5.2, moe, speculative-decoding, quantization, expert-streaming, consumer-hardware, c-runtime","platforms":"Claude, ChatGPT, Gemini, Codex, jeopi","version":"1.0","source":"JustVugg/colibri"}

Colibri

Assist with Colibri — the pure-C inference engine for running GLM-5.2 (744B Mixture-of-Experts) on consumer machines with ~25 GB RAM, featuring expert disk-streaming, MLA attention, and native speculative decoding (MTP).

When to use this skill

• Setup & build: Installing C build tools, running setup.sh, configuring OpenMP, cross-platform compilation (Linux, macOS, Windows via WSL)
• Model download & conversion: Running coli convert to fetch GLM-5.2-FP8 shards, convert to int4 container, prepare MTP heads (int8 quantization)
• Inference modes: Running chat mode (coli chat), API server setup, batch inference, configuring sampling and temperature
• Expert caching & storage: Tuning LRU cache sizes, pin-hot-store configuration, disk I/O optimization, thermal management
• Speculative decoding (MTP): Enabling/disabling multi-token prediction, measuring draft acceptance rates, optimizing verification latency
• GPU acceleration: Configuring CUDA pinned experts, GPU vRAM budgeting, hybrid CPU-GPU inference
• Performance profiling: Measuring tokens/second, analyzing expert-load patterns, memory residency tuning
• Integration & deployment: Embedding Colibri in applications, wrapping inference API, local-first LLM pipelines

When not to use this skill

• Python-first LLM frameworks: Use llama-index, crewai-multi-agent, or pydantic-ai for high-level abstractions
• Distributed/cloud inference: Use cloud LLM APIs or deepspeed for multi-GPU/multi-node training
• LLM fine-tuning: Use dspy, openrlhf-training, or moe-training for training workflows
• General AI infra: Use system-environment-setup for unrelated dev environments

Core architecture

Main components:

1. glm.c (~2,400 lines) — Faithful GLM-5.2 (glm_moe_dsa) forward pass with MLA attention, MTP speculative decoding, DeepSeek-V3 sigmoid router
2. Expert streaming — 21,504 routed experts (75 MoE layers × 256 experts) live on disk (~370 GB total at int4), streamed on demand with per-layer LRU cache
3. Storage hierarchy — Dense layers (attention, shared experts, embeddings) in RAM (~9.9 GB at int4), experts on NVMe/HDD, OS page cache as L2
4. MTP speculative decoding — GLM-5.2's native multi-token-prediction head (layer 78, int8 for lossless 39–59% draft acceptance) drafts and main model verifies in one forward
5. MLA attention — q/kv-LoRA + interleaved partial RoPE with compressed KV-cache (576 floats/token vs 32,768 dense — 57× smaller)

CLI modes:

• coli chat — Interactive chat with streaming, auto-detected RAM/cache/MTP
• coli convert — Download GLM-5.2-FP8, quantize to int4, prepare MTP int8 head
• coli plan — Inspect storage hierarchy, vRAM budget, expert placement before loading

Key parameters

Parameter	Scope	Default	Notes
COLI_MODEL	env	(required)	Path to converted int4 model directory
--ram <GB>	plan/chat	auto-detect	RAM budget for dense layers (min 10 GB)
--vram <GB>	plan/chat	0 (CPU-only)	GPU vRAM for pinned experts (requires CUDA build)
--cache <MB>	plan/chat	auto-detect	Per-layer LRU cache size for routed experts
--mtp on/off	chat	auto-detect	Enable/disable speculative decoding (off = single-token mode)
--temperature <0–2>	chat	0.95	Sampling temperature (0 = greedy)
--top-p <0–1>	chat	0.9	Nucleus sampling threshold

Performance metrics (community benchmarks)

• Tokens/second (CPU, int4 cache warm):
  • Single token: ~1.2 tok/s (minimal cache misses)
  • MTP enabled: ~2.2–2.8 tok/s (2.2–2.8× speedup, 39–59% draft acceptance)
  • MTP with cold cache: may regress due to extra expert loads (~660 → ~1100/token)
• Residency: Dense layers ~9.9 GB RAM; experts 370 GB disk
• Memory overhead: ~11 GB per-token token from routed experts; 57× smaller KV-cache than dense attention
• Conversion time: ~30–60 min (one-time, depends on disk speed and internet)

Integration patterns

• Local chat UI: Wrap coli chat in a terminal/web interface
• API server: Stream inference output over HTTP/gRPC
• Batch processing: Run multiple prompts with persistent model load
• Hybrid CPU-GPU: Pin hot experts to GPU vRAM, stream others from disk
• Edge/mobile: int4 model size (~370 GB) requires local NVMe; consider pruning/distillation for smaller devices