C/C++ LLM inference library with GGUF support, quantization, GPU acceleration (CUDA/Metal/HIP/Vulkan/SYCL), OpenAI-compatible server, and speculative decoding. Use when building local LLM inference applications, deploying models on edge devices, creating OpenAI-compatible API servers, or working with GGUF models.
Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.
Quelldateien prüfen
Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
C/C++ LLM inference library with GGUF support, quantization, GPU acceleration (CUDA/Metal/HIP/Vulkan/SYCL), OpenAI-compatible server, and speculative decoding. Use when building local LLM inference applications, deploying models on edge devices, creating OpenAI-compatible API servers, or working with GGUF models.
llama.cpp b8789
Overview
llama.cpp is a plain C/C++ implementation for LLM inference with minimal setup and state-of-the-art performance across a wide range of hardware — locally and in the cloud. It requires no external dependencies beyond system libraries, making it ideal for edge deployment, embedded systems, and resource-constrained environments.
Key capabilities:
GGUF model format — native support for GGUF files with extensive quantization options
Multi-backend GPU acceleration — CUDA (NVIDIA), Metal (Apple Silicon), HIP (AMD), Vulkan, SYCL (Intel), MUSA (Moore Threads), CANN (Ascend NPU)
OpenAI-compatible HTTP server — llama-server with full chat completions, embeddings, and tool calling support
Multimodal input — image and audio support via libmtmd for vision-language models
Speculative decoding — draft model and n-gram based acceleration
Function calling — OpenAI-style tool use with native format handlers for Llama 3.x, Hermes, Qwen, Mistral, and more
Grammar-constrained generation — GBNF grammars and JSON schema support for structured outputs
CPU optimizations — ARM NEON, AVX/AVX2/AVX512/AMX, RVV/ZVFH, ZenDNN, Arm KleidiAI
When to Use
Running LLM inference locally without cloud dependencies
Deploying models on edge devices with limited resources
Building OpenAI-compatible API endpoints for local model serving
Implementing multimodal AI (vision + text) applications
Quantizing large models for reduced memory footprint
Creating structured output pipelines with grammar constraints
Developing cross-platform AI applications (Windows, macOS, Linux, Android)
Building tools that integrate with LLMs via the libllama C API
Core Tools
llama-cli — Interactive CLI for conversation mode, completion, and experimentation. Supports chat templates, multimodal input, speculative decoding, grammar constraints, and all sampling parameters.
# Local model file
llama-cli -m my_model.gguf
# Download from Hugging Face
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF
# Multimodal model with image
llama-cli -hf ggml-org/gemma-3-4b-it-GGUF --image photo.jpg
llama-server — Lightweight OpenAI-compatible HTTP server for serving LLMs. Supports chat completions, embeddings, reranking, tool calling, multimodal input, router mode (multiple models), and streaming.
# Start server on port 8080
llama-server -m model.gguf --port 8080
# With Hugging Face model
llama-server -hf ggml-org/gemma-3-1b-it-GGUF
# Router mode (multiple models)
llama-server --models-dir ./my_models
llama-quantize — Convert GGUF models between quantization formats. Supports Q4_K_M, Q8_0, IQ2_XS, and many other schemes.
# Quantize to Q4_K_M
llama-quantize input-f32.gguf output-Q4_K_M.gguf Q4_K_M
# With importance matrix for quality
llama-quantize --imatrix imatrix.gguf input-f16.gguf output-Q4_K_M.gguf Q4_K_M
llama-perplexity — Measure model perplexity (quality metric) over text files.
llama-perplexity -m model.gguf -f test-corpus.txt
llama-bench — Benchmark inference performance across various parameters.
# Draft model (smaller model accelerates larger one)
llama-server -m large-model.gguf -md draft-model.gguf
# N-gram speculative decoding (no draft model needed)
llama-server -m model.gguf --spec-type ngram-simple --draft-max 64
Multimodal (Vision + Text)
# CLI with vision model
llama-mtmd-cli -hf ggml-org/gemma-3-4b-it-GGUF
# Server with multimodal support
llama-server -hf ggml-org/gemma-3-4b-it-GGUF
# Local files
llama-server -m text-model.gguf --mmproj mmproj.gguf
Advanced Topics
Building and Backends: CUDA, Metal, HIP, Vulkan, SYCL, CANN, ZenDNN, KleidiAI, OpenVINO build configurations and runtime tuning → Building and Backends
Model Quantization: Quantization formats (Q4_K_M, Q8_0, IQ2_XS, etc.), importance matrices, quality vs size tradeoffs, perplexity measurement → Model Quantization
Server API Reference: OpenAI-compatible endpoints (/v1/chat/completions, /v1/embeddings, /v1/completions), router mode, multimodal, function calling, streaming → Server API Reference
Multimodal and Speculative Decoding: Vision/audio models, libmtmd integration, draft model decoding, n-gram speculation, performance tuning → Multimodal and Speculative Decoding
Grammar-Constrained Generation: GBNF grammar format, JSON schema to grammar conversion, structured output patterns, LLGuidance integration → Grammar-Constrained Generation
libllama C API: Core data structures (llama_model, llama_context), context management, batch decoding, sampler chains, LoRA adapters, KV cache → libllama C API
