machinelearning-liftwing-inference-services

Triage Wikimedia ML inference-services incidents at a time T (now or past). TRIGGER on "investigate / triage / debug / post-mortem / what happened" when the subject is a LiftWing alert, ML-serve namespace, error spike, latency regression, or pod crash — including generic phrasing ("investigate this issue") when context (Phabricator task, pasted alert, log snippet) points at LiftWing, even with no alert string. Always trigger on: LiftWingServiceErrorRate, KubernetesDeploymentUnavailableReplicas, KubernetesPodCrashLooping; any namespace under revscoring*, revertrisk*, articletopic-outlink, or other ML-serve; clusters ml-serve-eqiad, ml-serve-codfw, ml-staging-codfw. SKIP only when modifying code without diagnosing a runtime incident. Pulls Prometheus via Grafana's anonymous datasource proxy; correlates with Gerrit merges in `inference-services` and `deployment-charts` (Phab IDs from `Bug:` footers). Read-only.

Analyze a model server's Python code and its deployment chart to identify performance bottlenecks and optimization opportunities for CPU and GPU (AMD MI300X, ROCm, vLLM) inference services on KServe. Use when you want to improve inference throughput or latency for a model.

Scaffold a new LiftWing ML service in operations/deployment-charts. Handles both adding to an existing namespace (append inference_services entry) and creating a brand-new namespace (full helmfile scaffold). Use when an engineer wants to deploy a new inference service for the first time.

Bump the Docker image tag for one or more LiftWing inference services in operations/deployment-charts after a new image is published by Jenkins. Use when a patch has merged in inference-services, PipelineBot has posted a new image tag on the Gerrit CL, and you need to update the corresponding values.yaml file(s) to deploy it.

Help pinpoint why a Wikimedia ML KServe/Knative InferenceService deployment is not working on ml-serve or ml-staging Kubernetes clusters.

Build and run a model server locally via Docker Compose, then test it with curl. Use when the engineer wants to test a model server locally before committing.

Scaffold a new KServe model server from scratch — create the model.py, Blubber config, Docker Compose service, pipeline config, and CI wiring. Use when the engineer wants to add a new inference service model to this repo.