vendor-ai-risk

name

vendor-ai-risk

description

Third-party AI vendor risk assessment skill. Evaluates AI vendors and tools across five dimensions: security, privacy, AI-specific risks, contractual protections, and regulatory compliance. Outputs a vendor risk scorecard with a go/no-go recommendation. Two modes: quick triage (5-minute assessment) and deep due diligence. Use when evaluating AI tools for procurement, vendor due diligence, or third-party AI risk assessment.

Vendor AI Risk

Skill Name: vendor-ai-risk
Version: 1.0.0
Category: GRC / Procurement
Maturity: Production

Overview

The Vendor AI Risk skill evaluates third-party AI vendors and tools for risk across five dimensions: security, privacy, AI-specific risks, contractual protections, and regulatory compliance. It outputs a vendor risk scorecard with a go/no-go recommendation and conditions. Two modes support different decision timelines: quick triage (5 minutes) and deep due diligence (full assessment).

Triggers

This skill activates on requests containing these keywords (pushy mode enabled):

• "AI vendor assessment"
• "evaluate AI tool"
• "AI vendor risk"
• "third party AI"
• "AI procurement"
• "should we use this AI tool"
• "vendor due diligence for AI"
• "vendor evaluation"
• "AI tool assessment"

Capabilities

Assessment Modes

Mode 1: Quick Triage (5-minute assessment)

Fast, high-level screening for red flags. Suitable for initial procurement filtering.

Input Required:

• Vendor name and AI product
• Intended use case (internal analytics, customer-facing, compliance-sensitive, regulated industry)
• Data sensitivity classification (public, internal, confidential, restricted)

Output:

• Go/no-go recommendation (Fast-Track, Review Needed, Not Recommended)
• Top 5 risk factors
• Critical questions for vendor
• Estimated cost and timeline for deep assessment

---

Mode 2: Deep Assessment (full due diligence)

Comprehensive evaluation across all risk categories. Suitable for strategic vendor partnerships or regulated use cases.

Input Required:

• Vendor responses to detailed due diligence questionnaire
• Security certifications (SOC 2, ISO 27001, etc.)
• Privacy documentation (DPA, data processing terms, sub-processor list)
• Model transparency information (model card, testing reports, bias audits)
• Proposed contract terms

Output:

• Detailed risk scorecard (0-100 per dimension, 0-100 overall)
• Go/no-go recommendation with conditions
• Required contractual amendments
• Monitoring and compliance checkpoints
• Recommended exclusions or data restrictions

---

Risk Assessment Dimensions

1. Security Assessment

Evaluation Criteria:

Factor	Assessment	Red Flags
Certifications	SOC 2 Type II (required)	No SOC 2, < 1 year in force
Encryption Standards	TLS 1.3 for transit; AES-256 at rest	TLS 1.2, unencrypted storage, no standards
Access Controls	MFA, RBAC, principle of least privilege	Shared passwords, no audit logging
Vulnerability Disclosure	Documented responsible disclosure program	No program, slow response (>90 days)
Incident Response SLA	< 4 hours detection, < 1 hour notification	No SLA, > 24 hour response
Penetration Testing	Annual third-party pen tests, remediation tracked	No pen tests, unclear remediation
Data Segregation	Customer data logically isolated, tenant isolation verified	Shared databases, co-location risks
Backup & Recovery	RTO < 4 hours, RPO < 1 hour, tested quarterly	No backups, untested recovery

Scoring:

• 90-100: Best-in-class security posture
• 70-89: Acceptable with monitoring
• 50-69: Multiple gaps, conditional approval
• <50: Critical gaps, not recommended

---

2. Privacy Assessment

Evaluation Criteria:

Factor	Assessment	Red Flags
Data Processing Agreement (DPA)	GDPR-compliant DPA provided	Refuses DPA, generic terms only
Training Data Use	Explicit: "No training on customer data"	Ambiguous, trains on data, opt-out only
Data Residency	Meets org requirements (EU/US/specific region)	Cross-border transfers, no residency control
Retention Limits	Auto-deletes per policy (default < 30 days)	Indefinite retention, unclear purge
Sub-processors	List provided, change notification 30+ days	No list, unannounced changes
Data Subject Rights	Supports GDPR: access, deletion, export	No API for rights requests, manual process
Breach Notification	Commits to <72 hour GDPR notification	Notification timeline vague
Consent Management	Consent collection & withdrawal tools	No mechanism, assumes consent
Right to Audit	Permits customer security audits annually	Audit access restricted, no API

Scoring:

• 90-100: Privacy-by-design, GDPR+ compliant
• 70-89: GDPR compliant with minor caveats
• 50-69: GDPR compliant but limited transparency
• <50: Privacy gaps, compliance risk

---

3. AI-Specific Risks

Evaluation Criteria:

Factor	Assessment	Red Flags
Model Transparency	Model card published (architecture, performance, limitations)	No documentation, "black box"
Bias & Fairness Testing	Third-party bias audit, fairness metrics disclosed	No testing, dismissed as "not applicable"
Hallucination Rate	Benchmarked hallucination rate published (<5% ideal)	Untested, vendor dismisses concern
Output Reliability	Factuality benchmarks, confidence scores in output	No confidence metrics, high variance
Model Versioning	Major version changes tracked, backwards compatibility tested	Silent model updates, breaking changes
Training Data Provenance	Training data composition documented (no copyrighted materials)	Unknown sources, copyright disputes
Adversarial Robustness	Tested against prompt injection, jailbreaks	No adversarial testing
Update Frequency	Clear schedule for security/performance patches	Ad-hoc, slow updates
Fine-tuning Options	Supports org-specific fine-tuning for accuracy	No customization, one-size-fits-all
Explainability	Output includes reasoning/confidence/uncertainty	Black-box scores without explanation

Scoring:

• 90-100: Highly transparent, well-tested, low hallucination risk
• 70-89: Transparent, some gaps in testing
• 50-69: Minimally transparent, hallucination risk acknowledged
• <50: No transparency, high hallucination risk

---

4. Contractual & Commercial

Evaluation Criteria:

Factor	Assessment	Red Flags
Liability for AI Outputs	Vendor assumes liability for inaccurate/biased outputs	Vendor disclaims all liability for outputs
Indemnification	Vendor indemnifies for IP infringement, regulatory violations	No indemnification clause
Service Level Agreement (SLA)	Uptime SLA >=99.5%, credits for breaches	No SLA, no service credits
Support Response	<1 hour for critical, <24 hours for high (24/7 availability)	Best-effort, no SLA
Audit Rights	Permits annual compliance audits, SOC 2 audit cooperation	Audit access denied
Termination Rights	30-day termination without cause, data deletion SLA	Long lock-in, data retention post-termination
Price Clarity	Transparent pricing, no hidden per-API-call charges	Opaque pricing, surprise overages
Insurance	E&O insurance $5M+, cyber liability $10M+	No insurance, low coverage
Data Ownership	Customer owns all data and model outputs	Vendor claims rights to outputs
Dispute Resolution	Arbitration available, appeal rights	Binding vendor-chosen jurisdiction

Scoring:

• 90-100: Vendor-friendly terms, comprehensive protections
• 70-89: Balanced terms with some vendor protections
• 50-69: Vendor-favorable terms, mitigation required
• <50: Unacceptable terms, recommend renegotiation or avoid

---

5. Regulatory Compliance

Evaluation Criteria by Jurisdiction:

Regulation	Assessment Criteria	Red Flags
EU AI Act	Vendor compliant with risk tier classification (high-risk requires conformity assessment)	Non-compliance, refuses assessment
GDPR	Compliant processor, DPA in place, adequate safeguards	No DPA, unlawful processing
HIPAA	(Healthcare) BAA in place, encryption, access controls	Refuses BAA, no security roadmap
PCI-DSS	(Payment cards) No cardholder data processing without Level 1 certification	Processes PAN without certification
FedRAMP	(US Gov) FedRAMP authorization if serving federal agencies	Not authorized, cannot serve federal use cases
SOC 2 Type II	(Financial/data services) Current Type II attestation	Only Type I, no history, or expired
CCPA	California consumer privacy compliance	Non-compliant data handling
PIPEDA	(Canada) Privacy compliance documented	Fails Canadian privacy standards
Industry Certification	ISO 27001 / ISO 27018 (if applicable)	No industry certifications

Scoring:

• 90-100: Exceeds regulatory requirements in all applicable jurisdictions
• 70-89: Compliant with applicable regulations
• 50-69: Compliant with major regs, gaps in secondary regs
• <50: Non-compliant, regulatory risk

---

Risk Scorecard Output

Summary

================================================================================
                     VENDOR AI RISK SCORECARD
================================================================================

Vendor: CloudAI Analytics Pro
Assessment Date: 2026-04-12
Assessment Mode: Deep
Assessed By: Security Team / Procurement

Use Case: Internal financial forecasting, non-sensitive data
Data Classification: Internal
Intended Deployment: Production

================================================================================
                         RISK DIMENSION SCORES
================================================================================

Security Assessment                          : 82/100 [ACCEPTABLE]
Privacy Assessment                           : 75/100 [ACCEPTABLE]
AI-Specific Risks                            : 68/100 [REVIEW NEEDED]
Contractual & Commercial                     : 71/100 [ACCEPTABLE]
Regulatory Compliance                        : 88/100 [STRONG]
                                              ____________
OVERALL RISK SCORE                           : 77/100 [GO WITH CONDITIONS]

================================================================================
                          RECOMMENDATION
================================================================================

STATUS: GO WITH CONDITIONS

Vendor is acceptable for internal use with the following conditions:
1. [CRITICAL] Require hallucination rate benchmark < 3% or add human review step
2. [HIGH] Amend contract: liability clause for biased outputs / indemnification
3. [HIGH] Obtain explicit opt-out from training data; add to DPA
4. [MEDIUM] Annual bias audit required; results shared with team

Estimated Start Date: 2026-05-01 (30 days for negotiation + onboarding)
Approval Gate: CIO + Legal sign-off after conditions documented

================================================================================
                       DETAILED FINDINGS
================================================================================

SECURITY [82/100]
✓ SOC 2 Type II current (expires 2027-06-30)
✓ TLS 1.3 + AES-256 encryption
✓ MFA enforced, RBAC implemented
✗ Incident response SLA: 8 hours (target <4 hours)
  Mitigation: Acceptable for non-critical internal use
✓ Annual pen tests with remediation tracking
✓ Tenant isolation verified in architecture review

PRIVACY [75/100]
✓ GDPR-compliant DPA template provided
✗ Data retention: 90 days default (want <30 days)
  Mitigation: Negotiate shorter default; API for manual deletion
✓ Sub-processor list current; 30-day change notification
✓ Supports data subject deletion requests
⚠ Training data: "not used, but may be used for model improvements"
  Mitigation: Add explicit opt-out and DPA amendment

AI-SPECIFIC RISKS [68/100]
✓ Model card published; architecture transparent
✗ Hallucination rate: Vendor reports 8% (want <3%)
  Mitigation: Add human review step for financial outputs
  Cost: 1-2 hours/week analyst time
✓ Bias testing: Third-party audit available, results accessible
✓ Confidence scores in output; uncertainty bounds provided
✗ Adversarial robustness: Not tested for prompt injection
  Mitigation: Acceptable for internal use; not customer-facing
✓ Fine-tuning available (cost: $5K one-time + $500/month)

CONTRACTUAL [71/100]
✓ Vendor assumes output liability for factually incorrect data
✗ Biased output liability: Vendor disclaims, offered shared liability
  Mitigation: Negotiate stronger indemnification
✓ 99.8% uptime SLA with service credits
✓ 24/7 support, <1 hour response for critical
✓ 30-day termination without cause
✓ Data deleted within 30 days of termination
✗ Insurance: E&O $3M (want $5M); cyber $5M (want $10M)
  Mitigation: Acceptable for non-critical financial forecasting

REGULATORY COMPLIANCE [88/100]
✓ EU AI Act: Vendor self-certifies non-high-risk
✓ GDPR compliant (DPA in place)
✓ SOC 2 Type II attestation
✓ No HIPAA/PCI-DSS required (internal non-regulated use)
⚠ ISO 27001 in progress (certification expected Q3 2026)

================================================================================
                   REQUIRED ACTIONS BEFORE APPROVAL
================================================================================

[CRITICAL PRIORITY]
- [ ] Obtain written commitment: hallucination rate <3% SLA
      Owner: Procurement | Due: 2026-04-15
- [ ] Amend contract liability clause to include biased/discriminatory outputs
      Owner: Legal | Due: 2026-04-20
- [ ] Add DPA amendment: no training use, explicit opt-out
      Owner: Legal + Vendor | Due: 2026-04-25

[HIGH PRIORITY]
- [ ] Negotiate data retention: 30-day default (from 90)
      Owner: Privacy | Due: 2026-04-18
- [ ] Request ISO 27001 interim attestation or third-party audit
      Owner: Security | Due: 2026-05-01

[MEDIUM PRIORITY]
- [ ] Fine-tune model on historical financial data (3-month pilot)
      Owner: Data Science | Due: 2026-05-15
- [ ] Schedule annual bias audit; add to compliance calendar
      Owner: Compliance | Due: 2026-05-01

================================================================================
                       ONGOING MONITORING
================================================================================

Quarterly Compliance Check (every 90 days):
- [ ] Verify data retention policy still <30 days
- [ ] Confirm no unauthorized model updates
- [ ] Review incident logs (if any)
- [ ] Confirm training data non-use

Annual Review (every 12 months):
- [ ] Obtain updated SOC 2 attestation
- [ ] Third-party bias audit results
- [ ] Cost-benefit analysis vs. competitive offerings
- [ ] Regulatory/legal changes requiring DPA update
- [ ] Executive assessment: Continue, renegotiate, or replace

Immediate Escalation Triggers:
- Incident involving customer data or compliance violation
- Unannounced model version change affecting output format/accuracy
- Security incident; vendor notification delay >72 hours
- Regulatory warning or audit finding
- Financial/viability concerns (acquisition, layoffs, etc.)

================================================================================

---

Quick Triage Output (5-minute assessment)

For initial screening:

VENDOR AI RISK - QUICK TRIAGE
Vendor: TinyML Startup Inc.
Date: 2026-04-12

USE CASE: Customer chatbot integration
DATA SENSITIVITY: Confidential customer messages

RED FLAGS IDENTIFIED (Top 5):
1. [CRITICAL] No SOC 2 certification; founded 2024
2. [HIGH] Training data policy: "May use for model improvement"
3. [HIGH] No published model card; "proprietary methodology"
4. [MEDIUM] No liability clause in standard terms
5. [MEDIUM] Incident response SLA > 24 hours

RECOMMENDATION: NOT RECOMMENDED (customer-facing + high data sensitivity)
RATIONALE: Too many material gaps for confidential data + customer exposure
           Recommend revisiting after SOC 2 certification + updated terms

ALTERNATIVE: Consider larger vendors (OpenAI, Anthropic) with better compliance posture
            OR use vendor only for internal, non-sensitive use cases

NEXT STEPS:
- Request SOC 2 roadmap and target date
- If available, start deep assessment in 12 months when certified
- Budget 2-3 weeks for full due diligence if they provide strong updates

ESTIMATED COST: 40 hours internal time for deep assessment (if revisited)

---

Integration Points

• policy-drafter skill: Vendor policy assessment informs vendor policy requirements
• audit-trail skill: Vendor assessments logged; approval decisions traceable
• config/org-config.yaml: Pulls org.risk_appetite, org.industry, org.jurisdictions to customize risk thresholds
• guardrails skill: Approved vendors integrated into guardrail allow-lists

---

Best Practices

✓ Conduct quick triage for all vendors before investing in deep assessment
✓ Involve Legal, Security, Privacy, and Finance in assessment
✓ Document all vendor commitments and amendments in DPA
✓ Build vendor assessments into annual risk management cycle
✓ Escalate immediately if vendor fails critical checks post-approval
✓ Plan vendor replacement strategy (contract should allow 30-day exit)
✓ Review competitive landscape annually (faster, cheaper, less risky options?)
✓ Share lessons learned across organization (incident database)

---

Prompt Guidance for LLM:

When assessing vendors, ask:

• What is the intended use case? (Internal analytics, customer-facing, compliance-critical?)
• What data sensitivity will be processed? (Public, internal, confidential, restricted?)
• Which regulations apply? (GDPR, HIPAA, PCI-DSS, industry-specific?)
• What is the organization's risk appetite? (Conservative, balanced, risk-tolerant?)
• Do we already have preferred vendors or contractual frameworks?
• What is the timeline for approval? (5-minute triage vs. full due diligence)

Then provide actionable, prioritized recommendations that balance risk, cost, and opportunity.