| name | product-ai-risk |
| description | Review AI-product safety, reliability, privacy, security, fairness, uncertainty, human oversight, fallback, monitoring, drift, latency, and cost risks. Use for AI launch gates, agent actions, high-impact automation, incident preparation, production failure analysis, or system feasibility review. |
Product AI Risk
Model how the full sociotechnical system can fail, detect failure early, limit exposure, and recover safely.
Inputs
- Users, affected non-users, workflow, and environment
- Model capabilities, data flows, tools, and external actions
- Decision impact and error tolerance
- Human oversight and operational ownership
- Evaluation, monitoring, rollback, and incident capabilities
- Regulatory, contractual, privacy, and policy constraints
Workflow
- Map the system boundary: inputs, data, model, retrieval, tools, people, actions, feedback, and downstream dependencies.
- Identify harms from wrong output, omission, delay, disclosure, manipulation, overreliance, disparate performance, and unauthorized action.
- Assess severity, likelihood, detectability, exposure, and reversibility by affected group.
- Define preventive controls: scope limits, permissions, data minimization, grounding, validation, confirmation, rate limits, and human approval.
- Define detective controls: quality slices, drift, data integrity, latency, cost, abuse, override, and incident signals.
- Design uncertainty communication, safe fallback, correction, appeal, and manual recovery.
- Set staged exposure, kill switch, rollback, and incident ownership.
- Establish launch blockers, accepted risks, residual risk owner, and review schedule.
- Escalate high-impact or regulated decisions to qualified legal, security, privacy, safety, and domain owners.
Output contract
Return system map, risk register, affected groups, controls, monitoring, human-oversight design, fallback and recovery plan, launch blockers, residual risk acceptance, owners, and review triggers.
Quality gate
- The review covers the workflow and organization, not only the model.
- Critical actions use least privilege and confirmation appropriate to impact.
- Distribution shift and data feedback loops are monitored.
- Safe failure is usable under real operating conditions.
- Residual risk has an accountable human owner.
Avoid
- A generic ethics checklist without system-specific failure paths
- Treating disclaimers as controls
- Human review without time, expertise, authority, or interface support
- Monitoring averages that hide harmed segments
- Launching without rollback because the model passed offline tests
Source grounding
Operational synthesis informed by uncertainty, trust, feedback, and responsible AI practices in Building AI-Powered Products and data quality, distribution shift, monitoring, latency, and feedback-loop principles in Designing Machine Learning Systems.
