enterprise-agent-ops-gcc

name	enterprise-agent-ops-gcc
description	Operate long-lived agent workloads with observability, security boundaries, and lifecycle management. Specialized for GCC tech companies (Saudi Arabia, UAE, Kuwait, Qatar, Bahrain, Oman) building production AI systems with enterprise requirements, compliance needs, and reliability standards.
origin	ECC + GCC Specialization
tier	professional
version	1

Enterprise Agent Ops for GCC — Operate AI Agents at Scale

Use this skill for cloud-hosted or continuously running agent systems that need operational controls beyond single CLI sessions.

Designed specifically for GCC companies operating production AI workloads with enterprise requirements, regulatory compliance, and high-reliability standards.

---

When to Use This (GCC Context)

Use this skill when:

• Running long-lived agent workloads (not single requests)
• You need production observability (logs, metrics, traces)
• You have compliance requirements (GCC regulatory)
• You need security boundaries and access controls
• You're operating agents at enterprise scale
• You need incident response and rollback capabilities
• You're building multi-tenant or mission-critical systems
• You need audit trails for regulatory compliance

Do NOT use when:

• Single request/response (use basic Claude API)
• Development/testing only
• No observability requirements
• No compliance or security needs

---

Four Operational Domains

Domain 1: Runtime Lifecycle Management

Managing agent workloads across their lifecycle

START
  ├─ Health check (API responsive?)
  ├─ Resource allocation (memory, compute)
  ├─ Dependency check (databases, APIs accessible?)
  └─ Warm-up (initialize state, load models)

PAUSE
  ├─ Graceful drain (finish in-flight requests)
  ├─ State snapshot (save to storage)
  ├─ Resource scaling down
  └─ Notification (track pause reason)

STOP
  ├─ Final state snapshot (critical for recovery)
  ├─ Close all connections
  ├─ Release resources
  └─ Audit log entry (why stopped, by whom)

RESTART
  ├─ Restore state from snapshot
  ├─ Verify data integrity
  ├─ Health check post-restart
  └─ Gradual traffic ramp-up

GCC-Specific Considerations:

REGIONAL DEPLOYMENTS
  - Saudi Arabia (main region): Primary agent pool
  - UAE (regional hub): Secondary agents, regional backup
  - Kuwait/Qatar: Smaller deployments or shared resources
  
COMPLIANCE IMPACT
  - SAMA (Saudi): Audit requirements for financial agents
  - DFSA (UAE): Data residency and transaction logging
  - CBK (Kuwait): Payment system uptime SLAs
  
OPERATIONAL WINDOWS
  - Business hours: 8 AM - 6 PM (local time per country)
  - Prayer times: Consider Jumu'ah (Friday noon) traffic patterns
  - National holidays: Each country different; affect load patterns

---

Domain 2: Observability (Logs, Metrics, Traces)

Deep visibility into what agents are actually doing

LOGS (Structured)
  {
    "timestamp": "2025-04-28T14:32:15Z",
    "agent_id": "agent-payroll-001",
    "user_id": "emp-5423",
    "region": "saudi-arabia",
    "action": "process_payroll",
    "status": "success",
    "duration_ms": 1234,
    "retry_count": 0,
    "cost_usd": 0.0045,
    "error": null,
    "audit_trail": "approved_by_hr_lead"
  }

METRICS (Tracked)
  - Success rate: % of agents completing assigned task
  - Mean retries: Average retries per task (target: <1.5)
  - Time to recovery: Minutes from failure detection to resolution
  - Cost per task: $ spent on API calls, compute (budget per country)
  - Error distribution: What % are transient vs. permanent failures
  - Latency: P50, P95, P99 response times

TRACES (Request Flow)
  User Request
    └─ Agent receives instruction (timestamp, region)
      └─ Agent calls external API (Saudi bank? UAE payment system?)
        └─ Process response (transform, validate)
          └─ Agent updates database (audit)
            └─ Return result to user (with audit ID)

ALERTS
  - Success rate drops below 95%
  - Latency exceeds SLA (e.g., 5 seconds)
  - Cost per agent-hour exceeds budget
  - Retry rate exceeds threshold
  - Compliance log gap detected

GCC-Specific Observability:

REGULATORY LOGGING (mandatory)
  - SAMA agents: Log every financial decision with justification
  - DFSA agents: Log data access, transformation, output
  - CBK agents: Log payment attempts with result codes

MULTI-COUNTRY DASHBOARDS
  - Saudi Arabia: Separate dashboard with SAMA-relevant metrics
  - UAE: Dashboard with DFSA compliance indicators
  - Kuwait/Qatar: Aggregated or separate per size

COST TRACKING BY REGION
  - Saudi Arabia: $X/agent-month budget
  - UAE: $Y/agent-month budget (often higher, more competition)
  - Kuwait: $Z/agent-month budget
  
  Track actual vs. budget, alert if approaching limit

---

Domain 3: Safety Controls (Scopes, Permissions, Kill Switches)

Preventing runaway or compromised agents

LEAST-PRIVILEGE CREDENTIALS

Agent Payroll (Saudi Arabia):
  ├─ Read access: Employee database, payroll rules
  ├─ Write access: Payroll ledger (ONE database, locked by date)
  ├─ API access: Saudi bank wire endpoint (no other banks)
  ├─ Cannot: Delete records, modify policies, access HR files
  └─ Rate limit: 100 payroll runs per day

Agent Customer Support (UAE):
  ├─ Read access: Customer tickets, knowledge base
  ├─ Write access: Ticket responses (append-only, no delete)
  ├─ API access: Email service (send only to customer)
  ├─ Cannot: Access other customers' data
  └─ Rate limit: 1000 responses per day, 100 per hour

SCOPE BOUNDARIES

Financial Agents:
  ├─ Amount boundaries: Max transaction $50,000 (per country)
  ├─ Time boundaries: Only execute during business hours
  ├─ User boundaries: Only for approved users in system
  ├─ Date boundaries: Transactions only in current month
  └─ Audit: Every action requires approval + logging

Operational Agents:
  ├─ System boundaries: Cannot modify code, configs
  ├─ Data boundaries: Cannot export user PII
  ├─ Cost boundaries: Cannot spend > daily budget
  ├─ Retry boundaries: Max 3 retries per action
  └─ Audit: Actions logged but not all require approval

KILL SWITCHES

Hard Kills (emergency stop):
  - Global kill switch: Stop ALL agents immediately
  - Region kill switch: Stop Saudi agents, keep UAE running
  - Agent kill switch: Stop one agent, keep others running
  - Feature kill switch: Disable risky feature, keep agent running

Implementation:
  - Kill switch: Simple flag in Redis (instant propagation)
  - Check interval: Every 10 seconds (max 10s to stop)
  - Verification: Agents acknowledge kill within 5 seconds
  - Rollback: Restore from last known good state

PERMISSION MODEL

Admin (CEO/CTO):
  ├─ Can: Start/stop agents, kill switches, view all logs
  ├─ Cannot: Modify agent code (must go through CI/CD)

Manager (Team lead):
  ├─ Can: Start/stop agents they own, view their logs, set budgets
  ├─ Cannot: Kill switch, view other teams' logs

Operations:
  ├─ Can: Monitor agents, respond to alerts, basic restarts
  ├─ Cannot: Deploy new agents, modify permissions

Agent (the AI itself):
  ├─ Can: Execute assigned tasks within scope
  ├─ Cannot: Exceed boundaries, escalate privileges, access other agents

GCC-Specific Safety Controls:

REGULATORY ENFORCEMENT
  - SAMA requirements: Mandatory approval before financial transactions
  - DFSA requirements: Data access audit trail, export controls
  - CBK requirements: Payment system validation before sending

AUDIT TRAIL (legally required in GCC)
  - Who approved the action? (name, ID, timestamp)
  - What was the action? (detailed description)
  - Why was it approved? (business justification)
  - When did it execute? (exact timestamp, timezone)
  - What was the result? (success/failure with details)
  
  Example SAMA entry:
    {
      "timestamp": "2025-04-28T14:32:15+03:00",
      "action": "transfer_funds",
      "amount_sar": 50000,
      "approved_by": "ahmed_alshammari (emp#5234)",
      "approval_time": "2025-04-28T14:31:00+03:00",
      "justification": "Employee bonus for Q1 2025",
      "executed_at": "2025-04-28T14:32:15+03:00",
      "result": "success",
      "transaction_id": "txn_2025_04_28_001"
    }

---

Domain 4: Change Management (Rollout, Rollback, Audit)

Deploying agent changes safely to production

DEPLOYMENT PIPELINE

1. DEV ENVIRONMENT
   ├─ Agent code changes
   ├─ Local testing (unit + integration)
   ├─ Cost estimation (how much will this cost?)
   └─ Security audit (permissions, scope, risk)

2. STAGING ENVIRONMENT
   ├─ Production-like data (real data if possible, otherwise sanitized)
   ├─ Load testing (10% of production traffic)
   ├─ Cost monitoring (validate estimates)
   ├─ Compliance check (SAMA/DFSA/CBK requirements met?)
   └─ Performance baseline (latency, error rate, cost)

3. CANARY ROLLOUT (production)
   ├─ Start with 1% of traffic (100 users? 1000 requests?)
   ├─ Monitor for 1 hour: error rate, latency, cost
   ├─ If all metrics green: 10% rollout
   ├─ If any red: STOP and investigate
   ├─ Gradual ramp: 10% → 25% → 50% → 100%
   └─ Each stage: 30 minutes minimum monitoring

4. FULL ROLLOUT
   ├─ 100% traffic on new agent version
   ├─ Continue monitoring for 24 hours
   ├─ Lock old version (can't be used anymore)
   └─ Keep backup for rollback (48 hours)

AUTOMATIC ROLLBACK

When to trigger:
  - Error rate exceeds 5% (vs. baseline <1%)
  - Latency P99 exceeds SLA by 50%
  - Cost per request exceeds budget by 20%
  - Compliance check fails (regulatory violation)
  - Kill switch activated by ops team

How to execute:
  1. Stop new version immediately
  2. Restore previous version from snapshot
  3. Verify health checks pass
  4. Route traffic back to previous version
  5. Notify stakeholders (ops, PM, compliance)
  6. Post-mortem: Why did this fail? (add test)
  7. Fix + retry

AUDIT TRAIL

Every deployment must record:

{ "deployment_id": "deploy_2025_04_28_001", "timestamp": "2025-04-28T10:00:00Z", "region": "saudi-arabia", "agent_name": "payroll-processor", "version_from": "1.2.3", "version_to": "1.2.4", "deployed_by": "ops_engineer_5", "approval_by": "tech_lead_3", "change_summary": "Fix: Handle negative salary edge case", "rollout_strategy": "canary 1% -> 10% -> 50% -> 100%", "monitoring_period_minutes": 120, "metrics": { "error_rate_baseline": 0.5, "error_rate_canary_1pct": 0.4, "error_rate_canary_10pct": 0.45, "final_error_rate": 0.48, "cost_baseline": 0.0045, "cost_new": 0.0048 }, "status": "success", "rollback_triggered": false, "notes": "Smooth rollout, all metrics green" }

GCC-SPECIFIC AUDIT
- SAMA agents: Deployment must be approved by compliance officer
- DFSA agents: Data change audit required before rollout
- CBK agents: Payment system validation before deployment

---

Baseline Controls (Mandatory)

All production agent systems must have:

1. IMMUTABLE DEPLOYMENT ARTIFACTS
   - Every version is tagged with git commit hash
   - No "latest" that can change; versions are fixed
   - Old versions archived (for rollback capability)
   - Example: agent-payroll:v1.2.3-abc123def456

2. LEAST-PRIVILEGE CREDENTIALS
   - API keys scoped to single endpoint (no master keys)
   - Database users with minimal permissions
   - Credentials rotated every 90 days
   - Example: Saudi agent has Saudi bank access, nothing else

3. ENVIRONMENT-LEVEL SECRET INJECTION
   - Secrets never in code or git history
   - Secrets injected at container runtime
   - Different secrets per region (Saudi ≠ UAE)
   - Example: 
     env BANK_API_KEY_SAUDI=$SECRET_SAUDI \
         BANK_API_KEY_UAE=$SECRET_UAE \
         agent-payroll start

4. HARD TIMEOUT AND RETRY BUDGETS
   - Every agent action has max timeout (e.g., 30 seconds)
   - Max retries per action (e.g., 3 retries)
   - Exponential backoff (1s, 2s, 4s, then fail)
   - Example:
     action.timeout = 30000  # 30 seconds
     action.max_retries = 3
     action.backoff = exponential(base=1)

5. AUDIT LOG FOR HIGH-RISK ACTIONS
   - Financial transactions: Logged with approval chain
   - User data access: Logged with justification
   - Permission changes: Logged with approver
   - Deployment: Logged with version, approver, result
   - Example: See audit trail section above

---

Metrics to Track

Always collect these metrics:

SUCCESS METRICS
  - Success rate: % of tasks completed successfully
    Target: >95%
    Alert: <90%
    
  - Mean retries per task: Average retries needed
    Target: <1.5
    Alert: >2.0
    
  - Time to recovery: Minutes from failure to resolution
    Target: <15 minutes
    Alert: >60 minutes

EFFICIENCY METRICS
  - Cost per successful task: Dollar amount
    Target: Varies by task (e.g., $0.005 for payroll)
    Alert: +20% above baseline
    
  - Latency (P50, P95, P99): Response time percentiles
    Target: P99 <5 seconds
    Alert: P99 >10 seconds

ERROR METRICS
  - Failure class distribution: What % are each type?
    - Transient (network hiccup, retry worked): 60%
    - Permanent (data invalid, won't retry): 30%
    - Configuration (agent misconfigured): 10%
    
  - Error rate: % of requests that failed
    Target: <1%
    Alert: >5%

OPERATIONAL METRICS
  - Agent uptime: % of time agent is running
    Target: >99.5%
    Alert: <99%
    
  - Deployment frequency: How often do we ship?
    Target: Daily deployments
    Alert: >1 week since last deployment
    
  - Mean time to detect (MTTD): How fast do we notice failures?
    Target: <1 minute
    Alert: >5 minutes
    
  - Mean time to recover (MTTR): How fast do we fix?
    Target: <15 minutes
    Alert: >1 hour

COMPLIANCE METRICS (GCC-specific)
  - Audit log completeness: % of actions logged
    Target: 100%
    Alert: <99.5%
    
  - Approval chain compliance: % with required approvals
    Target: 100%
    Alert: <99.9%
    
  - Data access violations: # of unauthorized accesses
    Target: 0
    Alert: Any violation triggers investigation

---

Incident Pattern: When Failure Spikes

Follow this workflow when things go wrong:

PHASE 1: IMMEDIATE RESPONSE (0-5 minutes)

1. FREEZE NEW ROLLOUT
   - Stop all new deployments
   - Prevent cascading failures
   - Command: kubectl rollout pause agent-payroll

2. ASSESS SEVERITY
   - Is it affecting customers? (severity 1)
   - Is it degrading performance? (severity 2)
   - Is it a minor issue? (severity 3)
   - Declare severity level

3. ACTIVATE INCIDENT RESPONSE
   - Notify on-call team
   - Open incident channel (Slack #incident-payroll)
   - Assign incident commander
   - Set 15-minute check-in cadence

PHASE 2: DIAGNOSIS (5-30 minutes)

4. CAPTURE REPRESENTATIVE TRACES
   - Get last 100 failed requests (full logs)
   - Get last 100 successful requests (to compare)
   - Extract common patterns in failures
   - Timeline: When did failure start?

5. ISOLATE FAILING ROUTE
   - Is it one customer? (isolated)
   - Is it one region? (regional)
   - Is it all users? (global)
   - What triggers the failure?

6. IDENTIFY ROOT CAUSE
   - Agent bug? (code issue)
   - Dependency failure? (database, API down)
   - Configuration? (wrong secrets injected)
   - Resource exhaustion? (out of memory)
   - External service? (bank API slow)

PHASE 3: REMEDIATION (30-90 minutes)

7. PATCH WITH SMALLEST SAFE CHANGE
   - One-line fix? (deploy immediately)
   - Bigger refactor? (use workaround instead)
   - Unknown root cause? (enable killing to stop impact)
   - Examples:
     * Change: Add null check → Deploy v1.2.5
     * Workaround: Disable feature → Use kill switch
     * Unknown: Rollback to v1.2.3 → Investigate later

8. RUN REGRESSION CHECKS
   - Does fix solve the symptom?
   - Does fix introduce new problems?
   - Test in staging if possible (5 min)
   - Manual spot check (3 min)

9. RUN SECURITY CHECKS
   - Did fix introduce vulnerabilities?
   - Are audit logs still working?
   - Are permissions still correct?
   - GCC compliance still met?

10. RESUME GRADUALLY
    - Start canary 1%: Monitor 5 minutes
    - Expand to 25%: Monitor 5 minutes
    - Expand to 50%: Monitor 5 minutes
    - Expand to 100%: Monitor 30 minutes

PHASE 4: RESOLUTION (90+ minutes)

11. VERIFY STABILITY
    - Success rate back to >95%?
    - No new error patterns?
    - Cost normalized?
    - Customers reporting success?

12. COMMUNICATE TO STAKEHOLDERS
    - What happened? (customer-friendly explanation)
    - Why did it happen? (root cause)
    - How did we fix it? (actions taken)
    - How do we prevent it? (systemic fix)

13. POST-MORTEM (within 24 hours)
    - Review incident logs
    - Document root cause
    - Add test to prevent recurrence
    - Update runbooks if needed

14. DEPLOY PERMANENT FIX
    - Go through normal deployment pipeline
    - Add regression test
    - Update documentation
    - Verify monitoring alerts are working

---

Deployment Integrations

Integration 1: PM2 Workflows (Single Server)

# ecosystem.config.js
module.exports = {
  apps: [
    {
      name: "agent-payroll-saudi",
      script: "./agent.js",
      instances: 2,
      exec_mode: "cluster",
      env: {
        NODE_ENV: "production",
        REGION: "saudi-arabia",
        AGENT_ID: "payroll-001"
      },
      max_memory_restart: "500M",
      error_file: "./logs/error.log",
      out_file: "./logs/out.log",
      log_date_format: "YYYY-MM-DD HH:mm:ss Z",
      watch: false,  // No auto-reload in production
      ignore_watch: ["node_modules", "logs"],
      max_restarts: 10,
      min_uptime: "10s"
    }
  ]
};

# In production:
pm2 start ecosystem.config.js
pm2 save           # Save state
pm2 startup        # Auto-restart on reboot
pm2 monit          # Monitor

Integration 2: Systemd Services (Linux)

# /etc/systemd/system/agent-payroll.service
[Unit]
Description=Payroll Agent (Saudi Arabia)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=agent
WorkingDirectory=/opt/agents/payroll
EnvironmentFile=/etc/agent-payroll.env
ExecStart=/usr/bin/node agent.js
Restart=on-failure
RestartSec=10
StandardOutput=journal
StandardError=journal

# Operational controls
TimeoutStopSec=30
KillMode=mixed

[Install]
WantedBy=multi-user.target

# Usage:
systemctl start agent-payroll
systemctl status agent-payroll
systemctl restart agent-payroll
journalctl -u agent-payroll -f

Integration 3: Kubernetes (Cloud-Native)

# agent-payroll-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: agent-payroll
  namespace: agents-saudi
  labels:
    agent: payroll
    region: saudi-arabia
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      agent: payroll
  template:
    metadata:
      labels:
        agent: payroll
        region: saudi-arabia
    spec:
      serviceAccountName: agent-payroll
      containers:
      - name: agent
        image: agents/payroll:v1.2.3-abc123
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8000
        env:
        - name: REGION
          value: "saudi-arabia"
        - name: BANK_API_KEY
          valueFrom:
            secretKeyRef:
              name: agent-secrets-saudi
              key: bank_api_key
        resources:
          requests:
            memory: "256Mi"
            cpu: "100m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 30
          periodSeconds: 10
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /ready
            port: 8000
          initialDelaySeconds: 5
          periodSeconds: 5

---
# agent-payroll-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: agent-payroll-service
  namespace: agents-saudi
spec:
  selector:
    agent: payroll
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8000
  type: ClusterIP

# Deployment:
kubectl apply -f agent-payroll-deployment.yaml
kubectl rollout status deployment/agent-payroll -n agents-saudi
kubectl rollout history deployment/agent-payroll -n agents-saudi
kubectl rollout undo deployment/agent-payroll -n agents-saudi  # Rollback

Integration 4: CI/CD Gates (GitHub Actions)

# .github/workflows/deploy-agent.yml
name: Deploy Agent to Production

on:
  push:
    branches: [main]
    paths:
      - 'agents/payroll/**'

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: npm install
      - run: npm run test
      - run: npm run security-scan
      - run: npm run cost-estimate
        
  staging:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: npm run deploy:staging
      - run: npm run smoke-test:staging
      - run: npm run compliance-check:staging
      
  approval:
    needs: staging
    runs-on: ubuntu-latest
    steps:
      - name: Request approval
        run: |
          echo "✅ Staging passed. Awaiting approval for production."
      
  production:
    needs: approval
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: npm run deploy:canary 1%
      - run: sleep 300 && npm run health-check
      - run: npm run deploy:canary 10%
      - run: sleep 300 && npm run health-check
      - run: npm run deploy:production 100%
      - run: npm run post-deployment-audit

---

GCC-Specific Operational Patterns

Pattern 1: Multi-Region Failover

Normal Operations:
  Saudi Arabia: Primary (100% traffic)
  UAE: Standby (0% traffic, up-to-date)
  Kuwait: Backup (0% traffic)

If Saudi agents fail:
  1. Detect failure (error rate >5% for 2 min)
  2. Activate failover: Route 50% to UAE, 50% to Saudi
  3. If Saudi still failing: 100% to UAE + Kuwait
  4. Notify ops team (auto-page on-call)
  5. Investigate Saudi (in parallel with running from UAE)
  6. Restore Saudi (data sync + health check)
  7. Gradual traffic shift back: 50% → 100%

Data consistency:
  - Use distributed transaction log
  - Saudi writes to log first (acknowledgment)
  - Then writes to local database
  - If Saudi fails before log update: UAE sees no write
  - If Saudi fails after log: UAE can complete write

Pattern 2: Cost Control Per Region

Budget Model:
  Saudi Arabia: $5,000/month (largest, Vision 2030 investment)
  UAE: $3,000/month (expensive but strategic)
  Kuwait: $1,000/month (smaller deployment)
  Qatar: $500/month (pilot)
  Total: $9,500/month

Tracking:
  - Daily cost report (8 AM, 2 PM, 8 PM local time)
  - Alert if daily burn > 1/25th of budget
  - Freeze new agent tasks if budget exhausted
  - Notify finance team weekly

Cost Optimization:
  - Batch similar requests (reduce API calls)
  - Cache results (avoid redundant processing)
  - Scheduled agents (off-peak hours for non-urgent)
  - Regional routing (use cheapest region if possible)

Pattern 3: Compliance Audit Readiness

Monthly Audit (SAMA Saudi):
  1. Extract all agent actions (SQL query)
  2. Verify 100% audit trail completeness
  3. Check all approvals documented
  4. Verify no unauthorized access
  5. Report to compliance team
  6. Store audit snapshot (immutable)

Quarterly Audit (DFSA UAE):
  1. Data lineage audit (where did data come from?)
  2. Data transformation audit (what changed?)
  3. Data export audit (where did it go?)
  4. Access control audit (who accessed?)
  5. Encryption audit (was data encrypted in transit/rest?)
  6. Report to DFSA

Annual Audit (External):
  1. Third-party security scan
  2. Penetration test
  3. Code review
  4. Operational procedures review
  5. Incident response test (simulate failure)
  6. Compliance certification update

---

Operational Runbook

Quick Reference (Emergency)

SITUATION: Agent is down, customers affected

IMMEDIATE (2 min):
  1. Check status: kubectl get pod -n agents-saudi | grep agent-payroll
  2. If down: kubectl delete pod agent-payroll-xxx -n agents-saudi
     (Kubernetes auto-recreates with fresh container)
  3. Check logs: kubectl logs agent-payroll-xxx -n agents-saudi
  4. Post status: "Incident #427 - Payroll agent down, ETA 5 min"

DIAGNOSIS (10 min):
  1. Check pod logs (full): kubectl logs -n agents-saudi agent-payroll-xxx --tail=1000
  2. Check resource usage: kubectl top node (is node out of memory?)
  3. Check dependencies: 
     - Saudi bank API reachable? (curl to endpoint)
     - Database responding? (SELECT 1 from prod_db)
     - Cache healthy? (redis-cli ping)
  4. Check recent deployments: Was something deployed in last hour?

REMEDIATION:
  If recent deployment:
    1. kubectl rollout undo deployment/agent-payroll -n agents-saudi
    2. Verify health: curl http://agent-payroll-service/health
    
  If dependency down:
    1. Restart dependency (e.g., redis restart)
    2. Verify agent recovers automatically (should within 10 sec)
    
  If resource exhaustion:
    1. Scale up: kubectl scale deployment agent-payroll --replicas=5
    2. Identify leak (memory? connections?) for tomorrow's fix
    
RECOVERY:
  1. Verify success rate >95%
  2. Post: "Incident #427 - Resolved. Action: [what we did]"
  3. Create postmortem ticket (for next day)
  4. Continue monitoring (30 min)

---

Quality Gate Checklist

Before deploying any agent to production:

CODE & TESTING
  □ Unit tests pass (>80% coverage)
  □ Integration tests pass (with real dependencies in staging)
  □ Load testing complete (can handle 2x expected load)
  □ Security scan clean (no vulnerabilities)

OPERATIONAL
  □ Logging configured (structured, with all key fields)
  □ Metrics instrumented (success rate, latency, cost)
  □ Alerts configured (error rate, latency, cost spike)
  □ Runbook written (what to do when things go wrong)

SAFETY
  □ Credentials scoped (only what agent needs)
  □ Timeout set (max 30 sec per action)
  □ Retry budget set (max 3 retries)
  □ Kill switch implemented (can stop immediately)

GCC COMPLIANCE
  □ Audit logging enabled (every action logged)
  □ Approval chain configured (who approves?)
  □ Data residency verified (correct region)
  □ Regulatory check passed (SAMA/DFSA/CBK)
  □ Encryption verified (in transit, at rest)

DEPLOYMENT READY
  □ Canary strategy planned (1% → 10% → 50% → 100%)
  □ Rollback plan documented (what to do if fails)
  □ Cost estimate provided (expected $ spend)
  □ On-call engineer assigned (who responds if fails?)

---

Integration with Other Skills

Pair with:

• Product Thinking for PMs — Validate before building the agent
• Research Ops — Gather evidence for operational decisions
• Token Budget Advisor — Control agent complexity and cost
• Knowledge Ops — Store operational playbooks for team
• Market Research GCC — Understand operational requirements per market

---

Summary: Enterprise Agent Ops for GCC

What: Operate long-lived agent workloads safely and reliably
When: Production agents needing enterprise controls
How: 4 domains (lifecycle, observability, safety, change management)
GCC Context: Compliance, multi-region, cost control, audit trails

Use this to run production agents your customers depend on.