| name | ai-ml-operate-model-monitoring |
| description | Use when operating a deployed model, prompt, retrieval system, or agent and monitoring quality, drift, safety, tools, latency, cost, provider changes, feedback, rollout, rollback, retraining, or retirement. |
Model Monitoring and AI Operations
Apply core observability, incident response, feedback, maintenance, and release skills. AI telemetry
adds behavior and data signals; it does not replace service health monitoring.
Monitor the behavioral release
Tag each trace with application feature, release, model/provider, prompt, tool schema, retrieval
index, guardrail, and experiment cohort versions. Track:
- Task success and calibrated quality proxies, segmented by workflow and material cohort.
- Safety, policy, permission, escalation, refusal, hallucination/grounding, and tool failure rates.
- Input quality, missingness, distribution change, retrieval coverage/freshness, and embedding/index
health.
- End-to-end and per-step latency, timeouts, retries, turns, tool calls, queueing, token/compute cost,
and provider errors.
- Human overrides, corrections, complaints, abandonment, and downstream outcome where available.
Do not log raw prompts, outputs, retrieved documents, or user identifiers by default. Sample and
redact under explicit privacy, access, retention, and review policy.
Detect quality change
Define baseline windows and thresholds before alerting. Drift is evidence to investigate, not an
automatic retraining command. Separate data drift, concept/task drift, provider/model behavior
change, prompt/retrieval regression, and product population change. Use delayed ground truth when
available; otherwise maintain validated proxy metrics and routine sampled human review.
Roll out and recover
Canary behavioral changes by cohort, shadow them when effects can be evaluated safely, compare
against the current baseline, and retain instant rollback of model, prompt, retrieval, tools, and
guardrails as a compatible unit. Provider fallbacks preserve contracts and safety or enter an
explicit degraded mode. Track provider deprecations and re-run evals before forced migrations.
Feed reviewed production failures into the eval dataset with provenance and privacy protection.
Retrain, re-index, revise prompts, or replace models only under a recorded trigger, evaluation,
approval, rollout, and rollback plan. Retire unused artifacts and data under retention policy.
Operational gate
An owner can identify the active behavioral versions, explain current quality/cost/latency/safety,
trace a failed run, detect broken inputs or provider drift, obtain feedback/ground truth, and roll
back without rebuilding the application.
