oraclecloud-incident-runbook

name	oraclecloud-incident-runbook
description	Self-service incident runbook for OCI outages — health probes, instance recovery, cross-AD/region failover. Use when OCI instances go down, the status page is silent, or you need automated recovery without waiting for support. Trigger with "oraclecloud incident", "oci outage runbook", "oci failover", "oci instance recovery".
allowed-tools	Read, Write, Edit, Bash(oci:*), Bash(python3:*), Grep
version	1.7.0
license	MIT
author	Jeremy Longshore <[e-mail]>
tags	["saas","oraclecloud","oci"]
compatibility	Designed for Claude Code

Oracle Cloud Incident Runbook

Overview

Self-service runbook for when OCI instances go down and the status page stays green. OCI's status page has a history of not acknowledging outages in real time (London Jan 2026 — 502s and instances disappearing for 10 minutes with no status update). OCI Support response times average 4+ hours for Sev-1 tickets. This runbook gives you health probes, automated instance recovery, cross-AD failover, and cross-region failover — all executable without waiting on Oracle.

Purpose: Detect OCI service degradation independently, recover instances automatically, and fail over to alternate availability domains or regions when the primary is impacted.

Prerequisites

• OCI CLI installed and configured — ~/.oci/config validated (see oraclecloud-install-auth)
• Python 3.8+ with the OCI SDK — pip install oci
• Pre-configured resources: at least one compute instance, a VCN with subnets in multiple ADs
• IAM policies: manage instances, manage volumes, inspect work-requests in the target compartment
• Boot volume backups enabled (recovery depends on having a recent backup)

Instructions

Step 1: Independent Health Probes

Do not trust the OCI status page alone. Run your own health checks against the OCI API:

import oci
import time

config = oci.config.from_file("~/.oci/config")

def probe_oci_health(config):
    """Probe OCI API endpoints independently of the status page."""
    results = {}

    # Probe 1: Identity service (lightest call)
    try:
        start = time.time()
        identity = oci.identity.IdentityClient(config)
        identity.list_regions()
        results["identity"] = {"status": "healthy", "latency_ms": int((time.time() - start) * 1000)}
    except oci.exceptions.ServiceError  e:
        results[] = {: , : (e.status)}

    
    :
        start = time.time()
        compute = oci.core.ComputeClient(config)
        compute.list_instances(compartment_id=config[], limit=)
        results[] = {: , : ((time.time() - start) * )}
     oci.exceptions.ServiceError  e:
        results[] = {: , : (e.status)}

    
    :
        start = time.time()
        network = oci.core.VirtualNetworkClient(config)
        network.list_vcns(compartment_id=config[], limit=)
        results[] = {: , : ((time.time() - start) * )}
     oci.exceptions.ServiceError  e:
        results[] = {: , : (e.status)}

     results

health = probe_oci_health(config)
 service, status  health.items():
    ()