ops-resilience

name	ops-resilience
description	Handling AWS control plane failures with time-limited retries and fallback approaches. Use when AWS API calls fail, timeout, or return 5xx errors. Covers retry strategies, control plane vs data plane distinction, and alternative approaches when the control plane is unresponsive.

AWS Control Plane Resilience

Detecting Control Plane Failures

Control plane calls are API operations that create, modify, describe, or delete resources. When the control plane is impaired, you'll see:

# Common error patterns indicating control plane issues
- HTTP 500/502/503/504 responses
- RequestTimeout / RequestExpired
- ServiceUnavailable
- InternalError / InternalFailure
- Throttling (may indicate overloaded control plane)
- Connection timeout / Connection refused
- "Unable to connect to the endpoint URL"

Quick Health Check

# Test if a region's control plane is responding
aws ec2 describe-regions --region <region> 2>&1
aws sts get-caller-identity 2>&1

# Check AWS Health Dashboard programmatically
aws health describe-events --filter "eventTypeCategories=issue" --region us-east-1

Retry Strategy

Standard Retry Pattern (use this first)

# Retry with exponential backoff: 3 attempts, 5s/15s/30s delays
# Total wait: ~50 seconds before giving up
MAX_RETRIES=3
RETRY_DELAY=5

for i in $(seq 1 $MAX_RETRIES); do
  result=$(aws <service> <operation> <params> --region <region> 2>&1)
  exit_code=$?

  if [ $exit_code -eq 0 ]; then
    echo "$result"
    break
  fi

  if [ $i -eq $MAX_RETRIES ]; then
    echo "FAILED after $MAX_RETRIES attempts: $result"
    echo "Control plane may be impaired. Switching to fallback approach."
    break
  fi

  delay=$((RETRY_DELAY * (2 ** (i - 1))))
  echo "Attempt $i failed. Retrying in ${delay}s..."
  sleep $delay
done

AWS CLI Built-in Retry Configuration

# Set retry mode and max attempts via environment
export AWS_RETRY_MODE=adaptive
export AWS_MAX_ATTEMPTS=5

# Or per-command
aws <service> <operation> --cli-read-timeout 30 --cli-connect-timeout 10

# Or in ~/.aws/config
# [default]
# retry_mode = adaptive
# max_attempts = 5

Time-Limited Retry (hard deadline)

Classify errors before retrying: 5xx errors and timeouts are retryable; 4xx errors (except 429 throttling) indicate client-side issues and should not be retried.

# Give up after TIMEOUT_SECONDS regardless of retry count
TIMEOUT_SECONDS=120
START_TIME=$(date +%s)
RETRY_DELAY=5
ATTEMPT=0

while true; do
  ATTEMPT=$((ATTEMPT + 1))
  ELAPSED=$(( $(date +%s) - START_TIME ))

  if [ $ELAPSED -ge $TIMEOUT_SECONDS ]; then
    echo "TIMEOUT: ${TIMEOUT_SECONDS}s exceeded after $ATTEMPT attempts."
    echo "Control plane unresponsive. Proceeding with fallback."
    break
  fi

  result=$(aws <service> <operation> <params> --region <region> 2>&1)
  if [ $? -eq 0 ]; then
    echo "$result"
    break
  fi

  delay=$((RETRY_DELAY < (TIMEOUT_SECONDS - ELAPSED) ? RETRY_DELAY : (TIMEOUT_SECONDS - ELAPSED)))
  echo "Attempt $ATTEMPT failed (${ELAPSED}s elapsed). Retrying in ${delay}s..."
  sleep $delay
  RETRY_DELAY=$((RETRY_DELAY * 2))
done

Control Plane vs Data Plane

Understanding this distinction is critical — the data plane often continues working when the control plane is down.

Service	Control Plane (may fail)	Data Plane (usually still works)
EC2	RunInstances, TerminateInstances, CreateImage	SSH/RDP to running instances, existing network traffic
Amazon RDS	CreateDBSnapshot, ModifyDBInstance, RestoreDBInstance	Database connections (read/write queries)
S3	CreateBucket, PutBucketPolicy	GetObject, PutObject (data operations)
Amazon ECS	UpdateService, RegisterTaskDefinition	Running tasks continue serving traffic
Amazon EKS	kubectl (via API server)	Running pods continue serving traffic
Lambda	CreateFunction, UpdateFunctionCode	Existing function invocations
Amazon EBS	CreateSnapshot, CreateVolume	Read/write to attached volumes

Fallback Approaches by Service

EC2: Control Plane Down

Cannot do: Create AMIs, take snapshots, launch instances, use MGN Can do: SSH/RDP into running instances, transfer data directly

Fallbacks:

# Instead of AMI/snapshot → use rsync between instances
sudo rsync -avzP -e "ssh -i key.pem" /data/ ec2-user@<target-ip>:/data/

# Instead of AMI/snapshot → use dd + coldsnap if EBS Direct APIs respond
# EBS Direct APIs are a separate endpoint from the EC2 control plane
coldsnap download snap-1234 disk.img

# Instead of SSM → use direct SSH if network path exists
ssh -i key.pem ec2-user@<ip>

# Windows: robocopy between instances
robocopy "C:\Data" "\\<target-ip>\C$\Data" /E /Z /MT:8

RDS: Control Plane Down

Cannot do: Create snapshots, modify instances, restore from snapshots Can do: Connect to database, run queries, dump data

Fallbacks:

# Instead of RDS snapshot → use native database dump
# Customer responsibility: Encrypt database connections with TLS/SSL, encrypt backup files at rest, secure S3 buckets (Block Public Access, SSE-KMS, TLS-only bucket policy, versioning, access logging), and manage backup file lifecycle.
# Use SSL/TLS for all connections and encrypt dump files before transfer

# MySQL (require SSL)
# Store credentials in ~/.my.cnf (chmod 600) or retrieve from AWS Secrets Manager
mysqldump -h <endpoint> -u <user> -p --ssl-mode=REQUIRED --single-transaction --all-databases | gzip > dump.sql.gz

# PostgreSQL (require SSL)
pg_dump -h <endpoint> -U <user> -Fc "dbname=<dbname> sslmode=require" > dump.dump

# SQL Server (via sqlcmd with encryption)
sqlcmd -S <endpoint> -U <user> -P "$SQLCMD_PASSWORD" -N -Q "BACKUP DATABASE [<db>] TO DISK='backup.bak' WITH COMPRESSION"

# Upload dump to S3 in healthy region (S3 data plane usually works)
aws s3 cp dump.sql.gz s3://<bucket-in-healthy-region>/ --sse aws:kms --region <healthy-region>

S3: Control Plane Down

Cannot do: Create buckets, modify policies, configure replication Can do: Read/write objects in existing buckets

Customer responsibility: Before using S3 for data plane fallback operations, configure the bucket with Block Public Access enabled, default encryption (SSE-KMS recommended), a bucket policy enforcing TLS-only access, versioning enabled for backup retention, and access logging configured. AWS is responsible for the S3 service infrastructure and enforcing the configured security policies.

Fallbacks:

# S3 data plane is highly resilient — GET/PUT usually work even during control plane issues
# If S3 in source region is completely down, check if cross-region replication was enabled
aws s3 ls s3://<replica-bucket>/ --region <healthy-region>

ECS/EKS: Control Plane Down

Cannot do: Update services, register task definitions, kubectl commands Can do: Running containers continue serving traffic

Fallbacks:

# ECS: running tasks keep running — no action needed for existing workloads
# For new deployments, you must wait for control plane recovery or deploy in another region

# EKS: if API server is down, you cannot manage pods
# Fallback: deploy in another region using exported manifests
kubectl --context <healthy-cluster> apply -f all-workloads.yaml

EBS: Control Plane Down

Cannot do: CreateSnapshot, CreateVolume, CopySnapshot Can do: Read/write to attached volumes, EBS Direct APIs (separate endpoint)

Fallbacks:

# EBS Direct APIs may still work when EC2 control plane is down
# Use coldsnap to download snapshots that already exist
coldsnap download snap-1234 disk.img

# Transfer the image out-of-band and re-upload in healthy region
coldsnap upload --wait disk.img

Decision Flow

When an AWS API call fails:

API call fails
  │
  ├─ Is it a transient error (throttle, timeout)?
  │   YES → Retry with exponential backoff (max 120s total)
  │          └─ Still failing? → Continue below
  │
  ├─ Is the data plane still working?
  │   YES → Use data plane fallback (native dumps, rsync, direct connections)
  │   NO  → Region may be fully impaired
  │
  ├─ Are other regions healthy?
  │   YES → Execute in healthy region (cross-region copy, rebuild)
  │   NO  → Escalate — open AWS Support case (Critical severity)
  │
  └─ Log all failure details: timestamp, error code, region, operation, fallback taken

Logging Failed Attempts

Log failures for post-incident review:

log_failure() {
  echo "[$(date -Iseconds)] FAILED region=$1 service=$2 operation=$3 error=$4 attempt=$5 fallback=$6" >> /tmp/aws-resilience.log
}