openvinotoolkit/model_server

Latest recorded source activity: October 8, 2026 at 10:58

SkillsMP catalog refreshed: October 29, 2026 at 18:30

skills collected: 1

GitHub stars: 951
GitHub forks: 283

Install with an AI assistant

Copy this prompt into the AI assistant you're using.

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/openvinotoolkit/model_server.

Repository README

Repository README source: README.md

Read the source README

Expand full README

# OpenVINO™ Model Server **High-performance model serving for Generative AI and classic deep learning — powered by [OpenVINO](https://github.com/openvinotoolkit/openvino) and optimized for Intel hardware.** [![Apache License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](https://github.com/openvinotoolkit/model_server/blob/main/LICENSE) [![Docker Pulls](https://img.shields.io/docker/pulls/openvino/model_server.svg)](https://hub.docker.com/r/openvino/model_server) [![GitHub Release](https://img.shields.io/github/v/release/openvinotoolkit/model_server)](https://github.com/openvinotoolkit/model_server/releases) [![Platform](https://img.shields.io/badge/platform-Linux%20%7C%20Windows-blue)](https://docs.openvino.ai/2026/model-server/ovms_docs_deploying_server.html) --- ## What is OVMS? OpenVINO Model Server (OVMS) is a production-grade, C++ inference server that exposes ML models over standard network APIs. It serves both **Generative AI**, **Agentic** workloads (LLMs, VLMs, image generation, audio) and **classic deep learning** models (object detection, classification, OCR, and more). - **OpenAI-compatible API** for text generation, embeddings, image generation, and audio - **KServe** APIs for classic model inference - **Runs anywhere** — Docker, bare metal, Kubernetes/OpenShift, Windows - **Intel-optimized** — CPU, GPU, NPU acceleration via OpenVINO ![OVMS diagram](docs/ovms_diagram.png) --- ## Quick Start ### Serve an LLM with OpenAI-compatible API **On Linux (Docker):** ```bash mkdir -p ${HOME}/models # Model is downloaded automatically from HuggingFace to models folder docker run --rm -p 8000:8000 \ --user $(id -u):$(id -g) -v ${HOME}/models:/models:rw \ openvino/model_server:latest \ --source_model OpenVINO/Qwen3-4B-int4-ov \ --model_repository_path /models \ --rest_port 8000 ``` > For GPU acceleration, use the `latest-gpu` image tag and pass `--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)` to expose the Intel GPU device. **On Windows (binary package):** ```bat mkdir c:\models ovms.exe --source_model OpenVINO/Qwen3-4B-int4-ov --model_repository_path c:\models --rest_port 8000 ``` **Query the model:** ```console pip install openai ``` ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") stream = client.chat.completions.create( model="OpenVINO/Qwen3-4B-int4-ov", messages=[{"role": "user", "content": "What are the 3 main tourist attractions in Paris?"}], stream=True, extra_body={"chat_template_kwargs": {"enable_thinking": False}} ) for chunk in stream: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="", flush=True) ``` > [LLM QuickStart](https://docs.openvino.ai/2026/model-server/ovms_docs_llm_quickstart.html) ### Serve a Classic Model with KServe API **Download the model:** ```console curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.bin -O curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.xml -O ``` **On Linux (Docker):** ```bash docker run --rm -d -u $(id -u) -v ${PWD}:/models -p 9000:9000 \ openvino/model_server:latest \ --model_name resnet --model_path /models/resnet50.xml \ --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" \ --port 9000 ``` > For GPU acceleration, use the `latest-gpu` image tag and pass `--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)` to expose the Intel GPU device. **Windows (binary package):** ```bat ovms --model_name resnet --model_path resnet50.xml --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" --port 9000 ``` Run inference with a sample client ```console pip install numpy tritonclient[grpc] curl -L -o image.jpeg https://github.com/openvinotoolkit/model_server/blob/main/demos/common/static/images/bee.jpeg?raw=true ``` ```python import numpy as np import tritonclient.grpc as grpcclient with open("image.jpeg", "rb") as f: image_bytes = f.read() client = grpcclient.InferenceServerClient(url="localhost:9000") inputs = [grpcclient.InferInput("image", [1], "BYTES")] inputs[0].set_data_from_numpy(np.array([image_bytes], dtype=object)) outputs = [grpcclient.InferRequestedOutput("output")] result = client.infer(model_name="resnet", inputs=inputs, outputs=outputs) output = result.as_numpy("output") # (1, 1000) FP32 print("Top-1 class index:", int(np.argmax(output[0]))) ``` > [Vision model QuickStart](https://docs.openvino.ai/2026/model-server/ovms_docs_quick_start_guide.html) --- ## Features ### Generative AI - [LLM text generation](https://docs.openvino.ai/2026/model-server/ovms_demos_continuous_batching.html) — continuous batching, streaming, structured output, speculative decoding - [VLM (Vision Language Models)](https://docs.openvino.ai/2026/model-server/ovms_demos_continuous_batching_vlm.html) - [AI Agents with MCP servers](https://docs.openvino.ai/2026/model-server/ovms_demos_continuous_batching_agent.html) - [Text embeddings](https://docs.openvino.ai/2026/model-server/ovms_demos_embeddings.html) — OpenAI-compatible `/v1/embeddings` - [Reranking](https://docs.openvino.ai/2026/model-server/ovms_demos_rerank.html) — Cohere-compatible API - [Image generation](https://docs.openvino.ai/2026/model-server/ovms_demos_image_generation.html) — OpenAI-compatible `/v1/images/generations` - [Speech recognition and TTS](https://docs.openvino.ai/2026/model-server/ovms_demos_audio.html) — OpenAI-compatible audio API - [GGUF model support](https://docs.openvino.ai/2026/model-server/ovms_demos_gguf.html) ### Classic Models & Pipelines - TensorFlow, ONNX, PaddlePaddle, OpenVINO IR model formats - [MediaPipe graphs](https://docs.openvino.ai/2026/model-server/ovms_docs_mediapipe.html) - [Python execution nodes](https://docs.openvino.ai/2026/model-server/ovms_docs_python_support_reference.html) - [Dynamic input shapes](https://docs.openvino.ai/2026/model-server/ovms_docs_dynamic_input.html) ### Deployment & Integration - [Docker](docs/deploying_server_docker.md), [bare metal (Linux & Windows)](docs/deploying_server_baremetal.md), [Kubernetes / OpenShift](docs/deploying_server_kubernetes.md) - [Model repository](https://docs.openvino.ai/2026/model-server/ovms_docs_models_repository.html): local storage, S3, GCS, Azure Blob, HuggingFace Hub - [Model versioning](https://docs.openvino.ai/2026/model-server/ovms_docs_model_version_policy.html) and [hot-reload](https://docs.openvino.ai/2026/model-server/ovms_docs_online_config_changes.html) - [Prometheus-compatible metrics](https://docs.openvino.ai/2026/model-server/ovms_docs_metrics.html) - [gRPC streaming](https://docs.openvino.ai/2026/model-server/ovms_docs_streaming_endpoints.html) - [C API](https://docs.openvino.ai/2026/model-server/ovms_docs_c_api.html) for embedding OVMS in native applications ### Hardware Acceleration - CPU (x86, including Xeon), Intel integrated and discrete GPU, NPU - See [supported accelerators](https://docs.openvino.ai/2026/model-server/ovms_docs_target_devices.html) [→ Full feature list](https://docs.openvino.ai/2026/model-server/ovms_docs_features.html) --- ## Documentation | Topic | Link | |---|---| | Deployment | [Deploying the server](https://docs.openvino.ai/2026/model-server/ovms_docs_deploying_server.html) | | Model repository | [Preparing models](https://docs.openvino.ai/2026/model-server/ovms_docs_models_repository.html) | | Client libraries | [Writing client code](https://docs.openvino.ai/2026/model-server/ovms_docs_server_app.html) | | Demos & examples | [Demos](https://docs.openvino.ai/2026/model-server/ovms_docs_demos.html) | | Release notes | [GitHub Releases](https://github.com/openvinotoolkit/model_server/releases) | --- ## Get the Server **Docker images**: ```text docker pull openvino/model_server:latest # Intel CPU docker pull openvino/model_server:latest-gpu # Intel CPU,GPU,NPU docker pull openvino/model_server:weekly # pre-production version with all accelerators enabled ``` - [Docker Hub](https://hub.docker.com/r/openvino/model_server) - [Red Hat Ecosystem Catalog](https://catalog.redhat.com/software/containers/intel/openvino-model-server/607833052937385fc98515de) **Binary official packages** (Linux & Windows): [GitHub Releases](https://github.com/openvinotoolkit/model_server/releases) Binary pre-production packages (Linux & Windows): [storage.openvinotoolkit.org](https://storage.openvinotoolkit.org/repositories/openvino_model_server/packages/weekly/) --- ## Contributing Contributions are welcome! Please open an issue or pull request on GitHub. See [security policy](security.md) for responsible disclosure. --- ## References - [OpenVINO Toolkit](https://software.intel.com/en-us/openvino-toolkit) - [Performance benchmarks](https://docs.openvino.ai/2026/about-openvino/performance-benchmarks.html) - [GenAI with CPU optimization — Intel whitepaper](https://cdrdv2-public.intel.com/864404/vFINAL_Intel%20SLM%20Whitepaper.pdf) - [RAG with OpenVINO Model Server — blog post](https://medium.com/openvino-toolkit/rag-building-blocks-made-easy-and-affordable-with-openvino-model-server-e7b03da5012b) - [AIPC turned into a mighty assistant](https://medium.com/openvino-toolkit/ai-pc-turned-into-a-mighty-ai-assistant-with-local-models-and-openvino-model-server-1f41913252c9) --- \* Other names and brands may be claimed as the property of others.

Loading the repository README…

Skills in this repository

classification pending

skill

occupation

description

updated

python-mcp-sdk-skill

unclassified

Guidance for implementing Python MCP SDK stdio servers with FastMCP, robust validation, and deterministic tool output.

2026-10-08

Showing 1 of 1 collected skill.