multi-tenant-llm-hosting

name	multi-tenant-llm-hosting
description	Design secure, multi-tenant LLM hosting platforms with tenant isolation, quotas, billing attribution, noisy-neighbor protection, and per-tenant policy controls.
license	MIT
metadata	{"author":"devops-skills","version":"1.0"}

Multi-Tenant LLM Hosting

Host many teams/customers on shared inference infrastructure without sacrificing security, performance, or cost governance.

When to Use This Skill

• Building an internal LLM platform shared by multiple teams
• Hosting LLM inference for external customers with isolation requirements
• Implementing per-tenant quotas, billing, and rate limiting
• Designing request routing for multi-model, multi-tenant environments
• Preventing noisy-neighbor issues on shared GPU infrastructure

Prerequisites

• Kubernetes cluster with GPU node pools
• API gateway or LLM gateway (LiteLLM, Envoy, Kong)
• Prometheus + Grafana for per-tenant observability
• Redis or equivalent for rate limiting state
• Billing system or cost attribution database

Isolation Model

• Strong tenant identity on every request
• Per-tenant API keys and scoped model access
• Namespace or workload isolation for high-risk tenants
• Strict data retention and log partitioning controls

vLLM Multi-Model Serving

# vllm-deployment.yaml - Multi-model serving with vLLM
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-gpt4o-equivalent
  namespace: llm-serving
  labels:
    app: vllm
    model-tier: premium
spec:
  replicas: 3
  selector:
    matchLabels:
      app: vllm
      model-tier: premium
  template:
    metadata:
      labels:
        app: vllm
        model-tier: premium
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.4.1
          args:
            - "--model=/models/llama-3.1-70b"
            - "--tensor-parallel-size=2"
            - "--max-model-len=8192"
            - "--gpu-memory-utilization=0.90"
            - 
             
          
              
               
              
               
          
            
               
               
               
            
               
               
               
          
              
               
               
      
          
          
             
      
          
           
           
      
         

 
 

   
   
  
     
     

   
  
    
       
       
  
    
      
         
         
    
      
          
           
          
             
             
             
             
             
          
              
               
              
               
          
            
               
               
               
            
               
               
               
          
              
               
               
      
          
          
             
      
          
           
           

Per-Tenant Quota Configuration

# tenant-quotas-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: tenant-quotas
  namespace: llm-serving
data:
  quotas.yaml: |
    tenants:
      acme-corp:
        tier: enterprise
        models_allowed:
          - llama-3.1-70b
          - llama-3.1-8b
          - nomic-embed-text
        rate_limits:
          requests_per_minute: 300
          tokens_per_minute: 500000
          concurrent_requests: 50
        budget:
          daily_limit_usd: 500.00
          monthly_limit_usd: 10000.00
          alert_threshold_percent: 80
        priority: high

      startup-xyz:
        tier: standard
        models_allowed:
          - llama-3.1-8b
          - nomic-embed-text
        rate_limits:
          requests_per_minute: 60
          tokens_per_minute: 100000
          concurrent_requests: 10
        budget:
          daily_limit_usd: 50.00
          monthly_limit_usd: 1000.00
          alert_threshold_percent: 80
        priority: medium

      internal-dev:
        tier: free
        models_allowed:
          - llama-3.1-8b
        rate_limits:
          requests_per_minute: 
           
           
        
           
           
           
         

Namespace Isolation for High-Risk Tenants

# tenant-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-acme-corp
  labels:
    tenant: acme-corp
    isolation: strict
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: tenant-isolation
  namespace: tenant-acme-corp
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: llm-gateway
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              name: llm-serving
      ports:
        - port: 8000
          protocol: TCP
    - to:
        - 
            
               
      
          
           

 
 

   
   

  
     
     
     
     
     
     

Request Routing and Rate Limiting

# gateway_router.py
"""Multi-tenant request router with rate limiting and model routing."""
import time
import json
import redis
from fastapi import FastAPI, HTTPException, Header, Request
from typing import Optional
import httpx
import yaml

app = FastAPI()
redis_client = redis.Redis(host="redis", port=6379, decode_responses=True)

# Load tenant config
with open("/etc/config/quotas.yaml") as f:
    TENANT_CONFIG = yaml.safe_load(f)["tenants"]

MODEL_ENDPOINTS = {
    "llama-3.1-70b": "http://vllm-gpt4o-equivalent:8000",
    "llama-3.1-8b": "http://vllm-economy:8000",
    "nomic-embed-text": "http://embedding-service:8000",
}

def check_rate_limit(tenant_id: str, config: dict) -> bool:
    """Check and update rate limit for a tenant."""
    key = f"ratelimit:{tenant_id}:{int(time.time() // 60)}"
    current = redis_client.incr(key)
    if current == 1:
        redis_client.expire(key, 120)
    return current <= config["rate_limits"]["requests_per_minute"]

def () -> :
    
    key = 
    current = (redis_client.get(key)  )
     current < config[][]

 () -> :
    
    key = 
    current_spend = (redis_client.get(key)  )
     current_spend < config[][]

 ():
    
    
    rates = {
        : {: , : },
        : {: , : },
        : {: , : },
    }
    rate = rates.get(model, {: , : })
    cost = (prompt_tokens * rate[] + completion_tokens * rate[]) / 

    
    spend_key = 
    redis_client.incrbyfloat(spend_key, cost)
    redis_client.expire(spend_key, )

    
    billing_key = 
    redis_client.rpush(billing_key, json.dumps({
        : time.time(),
        : model,
        : prompt_tokens,
        : completion_tokens,
        : cost,
    }))


  ():
    
     x_tenant_id   TENANT_CONFIG:
         HTTPException(status_code=, detail=)

    config = TENANT_CONFIG[x_tenant_id]
    body =  request.json()
    model = body.get(, )

    
     model   config[]:
         HTTPException(status_code=, detail=)

    
      check_rate_limit(x_tenant_id, config):
         HTTPException(status_code=, detail=)

    
      check_concurrent(x_tenant_id, config):
         HTTPException(status_code=, detail=)

    
      check_budget(x_tenant_id, config):
         HTTPException(status_code=, detail=)

    
    endpoint = MODEL_ENDPOINTS.get(model)
      endpoint:
         HTTPException(status_code=, detail=)

    
    concurrent_key = 
    redis_client.incr(concurrent_key)

    :
          httpx.AsyncClient(timeout=)  client:
            response =  client.post(
                ,
                json=body,
                headers={: },
            )
            result = response.json()

            
            usage = result.get(, {})
            record_usage(
                x_tenant_id, model,
                usage.get(, ),
                usage.get(, ),
            )

             result
    :
        redis_client.decr(concurrent_key)

Rate Limiting with Envoy

# envoy-ratelimit.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-ratelimit-config
  namespace: llm-serving
data:
  config.yaml: |
    domain: llm-gateway
    descriptors:
      # Per-tenant rate limits
      - key: tenant_id
        value: acme-corp
        rate_limit:
          unit: minute
          requests_per_unit: 300
      - key: tenant_id
        value: startup-xyz
        rate_limit:
          unit: minute
          requests_per_unit: 60
      - key: tenant_id
        value: internal-dev
        rate_limit:
          unit: minute
          requests_per_unit: 20

      # Global rate limit as safety net
      - key: global
        rate_limit:
          unit: second
          requests_per_unit: 100

Billing Integration

# billing_export.py
"""Export tenant usage data for billing systems."""
import redis
import json
from datetime import datetime, timedelta
from typing import Dict, List

redis_client = redis.Redis(host="redis", port=6379, decode_responses=True)

def generate_tenant_invoice(tenant_id: str, month: str) -> Dict:
    """Generate monthly invoice for a tenant."""
    billing_key = f"billing:{tenant_id}:{month}"
    records = redis_client.lrange(billing_key, 0, -1)

    usage_by_model = {}
    total_cost = 0.0
    total_requests = 0

    for record_json in records:
        record = json.loads(record_json)
        model = record["model"]

        if model not in usage_by_model:
            usage_by_model[model] = {
                "requests": 0,
                "prompt_tokens": 0,
                "completion_tokens": 0,
                "cost_usd": 0.0,
            }

        usage_by_model[model]["requests"] += 1
        usage_by_model[model]["prompt_tokens"] += record["prompt_tokens"]
        usage_by_model[model]["completion_tokens"] += record[]
        usage_by_model[model][] += record[]

        total_cost += record[]
        total_requests += 

     {
        : tenant_id,
        : month,
        : datetime.utcnow().isoformat(),
        : {
            : total_requests,
            : (total_cost, ),
        },
        : usage_by_model,
    }

 () -> :
    
    key = 
     (redis_client.get(key)  )

Noisy-Neighbor Controls

• Per-tenant RPM/TPM limits
• Concurrency caps and queue isolation
• Fair scheduling with weighted priority classes
• Backpressure and graceful degradation policies

# priority-classes.yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: tenant-enterprise
value: 1000
globalDefault: false
description: "Enterprise tenant workloads"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: tenant-standard
value: 500
globalDefault: false
description: "Standard tenant workloads"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: tenant-free
value: 100
globalDefault: false
description: "Free tier tenant workloads"

Per-Tenant Monitoring

# tenant-alerts.yaml
groups:
  - name: tenant-alerts
    rules:
      - alert: TenantBudgetWarning
        expr: |
          llm_tenant_daily_spend_usd
          / llm_tenant_daily_budget_usd > 0.80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Tenant {{ $labels.tenant }} at 80% of daily budget"

      - alert: TenantRateLimitHitting
        expr: |
          rate(llm_rate_limit_rejections_total[5m]) > 1
        for: 5m
        labels:
          severity: info
        annotations:
          summary: "Tenant {{ $labels.tenant }} hitting rate limits"

      - alert: TenantErrorRateHigh
        expr: |
          rate(llm_tenant_errors_total[5m])
          / rate(llm_tenant_requests_total[5m]) > 0.10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Tenant {{ $labels.tenant }} error rate above 10%"

Security Baseline

• Encrypt data in transit and at rest.
• Disallow cross-tenant cache leakage.
• Restrict debug data access by role.
• Audit all privileged administrative actions.

Operational Runbook

1. Onboard tenant with policy template.
2. Issue virtual key and quota profile.
3. Validate observability and billing tags.
4. Run tenant-specific load/safety tests.
5. Enable production traffic with canary limits.

Troubleshooting

Symptom	Check	Fix
Tenant getting 429 errors	Rate limit counters in Redis	Increase RPM/TPM limits or upgrade tier
One tenant slowing others	Concurrent request counts per tenant	Reduce concurrency cap for offending tenant
Billing data missing	Redis billing keys and export job logs	Check billing export CronJob and Redis connectivity
Tenant cannot access model	Tenant config in ConfigMap	Add model to models_allowed list
Cross-tenant data leakage	Cache key prefixes and namespace isolation	Ensure cache keys include tenant_id prefix
Budget alerts not firing	Prometheus scrape targets and alert rules	Verify metric export and Alertmanager config

Related Skills

• llm-gateway - Key management and traffic routing
• llm-cost-optimization - Cost controls and optimization tactics
• zero-trust - Identity-centric network and access patterns
• gpu-kubernetes-operations - GPU cluster management
• llm-inference-scaling - Autoscaling inference workloads