| name | ai-lead-ops |
| description | Guides AI ops leadership—LLM SRE, model/prompt releases, eval/incidents, cost/capacity, vendors, and
cross-functional cadence. Use for AI platform ops, LLM SLAs, incidents, rollout governance, unit
economics, red-team/eval gates, and team rituals—not memory (ai-memory-developer), context code
(ai-context-engineer), security programs (cybersecurity), token roadmaps
(ai-token-improvement-plan-engineer), solution architecture (applied-ai-architect-commercial-enterprise),
skills portfolio (ai-skill-manager), or vertical AI product eng management
(engineering-manager-vertical-ai-products).
Prompt/eval team management and golden-set release policy: engineering-manager-agent-prompts-evals.
Safeguard inference platform: ml-infrastructure-engineer-safeguards. Safeguard model research:
ml-research-engineer-safeguards.
|
AI Lead Ops
When to Use
- Standing up AI platform operations and production service reliability
- Defining SLAs/SLOs for LLM-powered features
- Running AI incident reviews and post-mortems
- Governing model, prompt, and index rollouts with tiered gates
- Tracking AI unit economics (cost per session, tokens per feature)
- Coordinating red-team and evaluation gates before releases
- Building team rituals and cadence across engineering, research, risk, and product
- Managing AI vendor relationships, contracts, and bake-offs
When NOT to Use
- Implementing memory stores or context packing code →
ai-memory-developer / ai-context-engineer
- Building RAG pipelines or agent tools →
ai-engineer
- Designing corporate AI policy or regulatory mapping →
ai-risk-governance
- General network penetration testing or enterprise security programs →
cybersecurity
- Structured token/cost improvement roadmaps with backlog →
ai-token-improvement-plan-engineer
- Commercial/enterprise AI solution architecture →
applied-ai-architect-commercial-enterprise
- Vertical AI product engineering managers and squad roadmaps →
engineering-manager-vertical-ai-products
Related skills
| Need | Skill |
|---|
| Build RAG, agents, eval harnesses | ai-engineer |
| Memory and context implementation | ai-memory-developer, ai-context-engineer |
| Risk tiering and policies | ai-risk-governance |
| Adversarial testing execution | ai-redteam |
| CI/CD and platform incidents | devops |
| Pipeline security | devsecops |
| Token optimization roadmap and initiative backlog | ai-token-improvement-plan-engineer |
| Commercial/enterprise AI architecture | applied-ai-architect-commercial-enterprise |
| Skills portfolio governance | ai-skill-manager |
| Safeguard inference platform | ml-infrastructure-engineer-safeguards |
| Safety classifier research | ml-research-engineer-safeguards |
Core Workflows
1. Operating model and cadence
| Ritual | Frequency | Outcomes |
|---|
| AI ops standup | Daily | Blockers, incidents, deploys |
| Model/prompt change review | Per release | Approvers, eval delta |
| Cost review | Weekly | Spend vs budget, top features |
| Risk & safety sync | Bi-weekly | Incidents, policy gaps |
| Quarterly capacity | Quarterly | Model roadmap, vendor contracts |
Define RACI: who owns model, prompt, index, eval suite, on-call.
See references/operating_model.md for roles and escalation.
2. Release governance
Production promotion checklist:
See references/release_governance.md for tiered gates and canary metrics.
3. SLOs, incidents, and observability
Example SLIs:
| SLI | Notes |
|---|
| Availability | Successful completion / total requests |
| Latency | p95 end-to-end |
| Quality proxy | Thumbs-down rate, escalation rate |
| Safety | Policy violation rate post-deploy |
| Cost | USD per successful session |
AI incident types: toxic output, PII leak in logs, retrieval cross-tenant leak, runaway agent loop, vendor outage.
See references/incidents_slos.md for severity matrix and post-incident template.
4. Cost and capacity
- Track tokens by model, feature, tenant
- Set budgets and alerts at 80/100/110%
- Optimize via routing, caching, context engineering (partner with
ai-context-engineer)
- Forecast from usage growth + model price changes
See references/cost_capacity.md for unit economics worksheet.
5. Vendor and eval program
- Maintain scorecard: quality, latency, safety, price, data terms
- Run structured bake-offs before annual renewals
- Own central eval harness ownership and dataset hygiene
See references/vendor_eval_program.md for RFP topics and eval program maturity.
When to load references
- Team cadence and RACI →
references/operating_model.md
- Releases and canaries →
references/release_governance.md
- SLOs and incidents →
references/incidents_slos.md
- Cost and capacity →
references/cost_capacity.md
- Vendors and eval ops →
references/vendor_eval_program.md
