llmops-platform-engineering

name	llmops-platform-engineering
description	Build production LLMOps platforms with CI/CD, model promotion workflows, evaluation gates, rollback, and governance across cloud and self-hosted inference.
license	MIT
metadata	{"author":"devops-skills","version":"1.0"}

LLMOps Platform Engineering

Design and operate an internal LLM platform that supports rapid experimentation without compromising reliability, cost, or compliance.

When to Use This Skill

• Building an internal platform for teams to deploy and manage LLM-powered features
• Designing CI/CD pipelines that include model evaluation gates
• Setting up A/B testing infrastructure for model versions
• Creating Kubernetes-based model serving infrastructure
• Establishing governance workflows for model promotion

Prerequisites

• Kubernetes cluster with GPU node pools (or cloud inference API access)
• Container registry (Harbor, ECR, GCR, or ACR)
• CI/CD system (GitHub Actions, GitLab CI, or Argo Workflows)
• Observability stack (Prometheus + Grafana + OpenTelemetry)
• Model registry (MLflow or custom metadata store)

Outcomes

• Standardized path from experiment to production
• Safe model rollout with quality and safety gates
• Repeatable infra modules for inference, vector DB, and observability
• Clear ownership model across platform, app, and security teams

Reference Architecture

1. Control Plane: model registry, prompt/version catalog, policy checks, eval pipeline.
2. Data Plane: inference gateway, vector database, cache, feature store.
3. Ops Plane: telemetry, alerting, SLO dashboards, cost analytics.
4. Security Plane: IAM boundaries, secret rotation, content filters, audit logs.

Model Promotion Pipeline

# .github/workflows/model-promotion.yaml
name: Model Promotion Pipeline
on:
  workflow_dispatch:
    inputs:
      model_name:
        description: "Model identifier"
        required: true
      model_version:
        description: "Model version to promote"
        required: true
      target_env:
        description: "Target environment"
        required: true
        type: choice
        options: [staging, production]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run quality evaluation suite
        run: |
          python -m evals.run \
            --model "${{ inputs.model_name }}:${{ inputs.model_version }}" \
            --suite quality \
            --output results/quality.json

      - name: Run safety evaluation suite
        run: 
          
         
           
         
           
         
           
         
          
         
        
             
           

  
     
     
       
    
         
         
  
     
     
    
        

         
         
             
         
           
         

Evaluation Gate Thresholds

# thresholds/quality.yaml
gates:
  groundedness:
    metric: groundedness_score
    min: 0.85
    comparison: gte
  task_success:
    metric: task_success_rate
    min: 0.90
    comparison: gte
  hallucination:
    metric: hallucination_rate
    max: 0.08
    comparison: lte
  regression:
    metric: quality_delta_vs_baseline
    min: -0.02
    comparison: gte
    description: "Must not regress more than 2% vs current production"

# thresholds/latency.yaml
gates:
  p50_latency:
    metric: latency_p50_ms
    max: 800
    comparison: lte
  p95_latency:
    metric: latency_p95_ms
    max: 2000
    comparison: lte
  p99_latency:
    metric: latency_p99_ms
    max: 5000
    comparison: 
  
     
     
     

A/B Testing Configuration

# ab-test-config.yaml
apiVersion: gateway.ai/v1
kind: ABTest
metadata:
  name: model-comparison-q1
  namespace: ai-production
spec:
  duration: 7d
  traffic_split:
    control:
      model: gpt-4o-2024-08-06
      weight: 70
    treatment:
      model: gpt-4o-2025-01-15
      weight: 30
  metrics:
    primary:
      - task_success_rate
      - user_satisfaction_score
    secondary:
      - latency_p95
      - cost_per_request
      - hallucination_rate
  guardrails:
    auto_rollback_if:
      - metric: task_success_rate
        threshold: 0.80
        window: 1h
      - metric: hallucination_rate
        threshold: 0.15
        window: 30m
  assignment:
     
     

Kubernetes Model Serving Deployment

# model-serving-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference
  namespace: ai-production
  labels:
    app: llm-inference
    model: gpt-4o
    version: "2025-01"
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: llm-inference
  template:
    metadata:
      labels:
        app: llm-inference
        model: gpt-4o
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
           
          
            
               
      
          
           
          
             
             
             
             
          
              
               
              
               
          
            
               
               
               
            
               
               
               
          
            
               
               
             
             
          
            
               
               
             
             
          
              
               
               
              
               
      
          
          
             
          
          
             
      
          
           
           
      
         

 
 

   
   

  
     
  
      
       
       
      
       
       

 
 

   
   

  
     
     
     
   
   
  
      
      
        
           
        
           
           
      
      
        
           
        
           
           
  
    
       
      
          
           
           
    
       
      
          
           
           

CI/CD Design for AI Services

• Build immutable containers with pinned dependencies and model hashes.
• Use environment promotion: dev -> stage -> prod.
• Fail deployment if:
  • regression evals drop below baseline,
  • safety tests exceed risk threshold,
  • p95 latency exceeds SLO budget.
• Store deployment evidence for audits (commit SHA, eval report, approver).

Operational SLOs

Signal	Target	Measurement Window
Availability	99.9%	30-day rolling
p95 Latency	< 1200ms	5-min buckets
Cost per request	< $0.05	1-hour average
Task success rate	> 90%	24-hour rolling
Groundedness	> 85%	24-hour rolling

Platform Guardrails

• Enforce tenant quotas and model allow-lists.
• Require structured output contracts for automation paths.
• Default to low-risk model settings for critical workflows.
• Disable unconstrained tool execution in production.

Tooling Stack (Example)

Layer	Tools
Orchestration	Argo Workflows, GitHub Actions, Airflow
Model Registry	MLflow, custom metadata DB
Gateway	LiteLLM, Envoy-based API gateway
Observability	OpenTelemetry + Prometheus + Grafana + Langfuse
Policy	OPA/Rego for deployment and runtime checks
Evaluation	RAGAS, custom eval harness, Promptfoo
Serving	vLLM, TGI, Triton Inference Server

Troubleshooting

Issue	Diagnosis	Resolution
Canary fails quality gate	Compare eval results with baseline	Adjust model config or revert version
Deployment stuck in rollout	Check pod events and resource quotas	Fix resource limits or node availability
A/B test shows no significant difference	Verify traffic split and sample size	Extend test duration or increase treatment weight
Model cold start too slow	Large model weight download	Use pre-cached PVCs or init containers
Eval pipeline flaky	Non-deterministic model outputs	Set temperature=0 for evals, increase sample size

Related Skills

• ai-pipeline-orchestration - Orchestrate ingestion and inference workflows
• agent-evals - Build evaluation gates for releases
• llm-gateway - Route and control LLM traffic
• model-registry-governance - Model lifecycle and approval workflows
• ai-sre-incident-response - AI-specific incident response