eks-ops

name	eks-ops
description	EKS and OpenClaw operations troubleshooting guide. Use when diagnosing pod issues, Discord connectivity problems, configuration changes, OpenClaw version updates, or any infrastructure operations on the your-opc EKS cluster.

EKS Operations Guide

Cluster Info

• EKS 1.35, Region: us-west-2, Cluster: your-opc
• Namespace: opc-agents
• All nodes ARM64 (Graviton c6g/m6g), NodePool forced arm64
• Storage: gp3 via ebs.csi.eks.amazonaws.com (Auto Mode native CSI)
• Auth: Bedrock via EKS Pod Identity (not IRSA)
• OpenClaw: v2026.3.28 (image digest-pinned)
• Operator: v0.23.0

OpenClaw Version Upgrade SOP

ALL upgrades are MANUAL and user-initiated. NEVER auto-upgrade.

Image digest is managed exclusively via kubectl patch. Git manifests do NOT contain digest — kubectl apply will not overwrite it (3-way merge preserves third-party fields).

Step 1: Check for updates

kubectl logs josh-0 -n opc-agents -c openclaw --tail=20 | grep "update available"

Report current version and available version to user. Do NOT proceed without explicit user approval.

Step 2: Canary deployment (Josh or Luna)

# Clear digest + set Always to pull new image
kubectl patch openclawinstance josh -n opc-agents --type merge \
  -p '{"spec":{"image":{"digest":"","pullPolicy":"Always"}}}'
kubectl delete pod josh-0 -n opc-agents

# Wait for pod Ready, then record new digest
kubectl get pod josh-0 -n opc-agents -o jsonpath='{.status.containerStatuses[?(@.name=="openclaw")].imageID}'

# Immediately pin canary to new digest
kubectl patch openclawinstance josh -n opc-agents --type merge \
  -p '{"spec":{"image":{"digest":"sha256:<new>","pullPolicy":"IfNotPresent"}}}'

Step 3: Soak period (default 24 hours, user can override)

Report to user:

"Canary josh is running v. Soak ends at . I will check health periodically."

During soak, check every few hours:

• Pod stays 3/3 Ready with 0 restarts
• Discord agent responds to @mention
• No "Config observe anomaly" loop in logs
• No OOM or heap errors
• Memory stability: kubectl top pod josh-0 -n opc-agents

If any issue during soak -> ABORT immediately (see Rollback).

Step 4: Soak complete

Report to user:

"Canary josh ran v for with no issues. Ready to roll out?"

Wait for user approval before continuing.

Step 5: Gradual rollout (one at a time, 30s+ interval)

Order: Luna -> Milo -> Rex -> Admin (yourself LAST)

For each agent:

kubectl patch openclawinstance <name> -n opc-agents --type merge \
  -p '{"spec":{"image":{"digest":"sha256:<new>","pullPolicy":"IfNotPresent"}}}'
kubectl delete pod <name>-0 -n opc-agents
# Verify: Ready + Discord "client initialized" in logs before next

Step 6: Before updating yourself (Admin)

Tell user:

"Updating myself now. New digest: sha256:. If I don't respond in 5 min, ask Rex to check: kubectl exec devlead-0 -n opc-agents -c openclaw -- sh -c 'kubectl get pod admin-0 -n opc-agents'"

Rollback

# Revert to previous digest
kubectl patch openclawinstance <name> -n opc-agents --type merge \
  -p '{"spec":{"image":{"digest":"sha256:<old>","pullPolicy":"IfNotPresent"}}}'


# Delete pod for clean restart
kubectl delete pod <name>-0 -n opc-agents

Report failure to user. Do NOT continue rollout.

Operator Update (Helm)

Cluster-level operation. Ask user to run:

helm upgrade openclaw-operator oci://ghcr.io/openclaw-rocks/charts/openclaw-operator \
  --namespace openclaw-system -f values-eks.yaml

Discord Channel Configuration

Add/Remove Channels (Persistent via CR patch, hot-reloaded)

kubectl patch openclawinstance <name> -n opc-agents --type merge \
  -p '{"spec":{"config":{"raw":{"channels":{"discord":{"guilds":{"<GUILD_ID>":{"channels":{"<CH_ID>":{"requireMention":false}}}}}}}}}}'

Discover channels

curl -s -H "Authorization: Bot $DISCORD_BOT_TOKEN" \
  "https://discord.com/api/v10/guilds/$GUILD_ID/channels" \
  | python3 -c "import sys,json; [print(f'{c[\"name\"]} id={c[\"id\"]}') for c in json.load(sys.stdin) if c['type']==0]"

Critical Gotchas

Discord

• guilds.<ID>.channels is a whitelist — unlisted channels stop responding
• All 3 Privileged Gateway Intents must be enabled per bot
• NEVER restart multiple bots simultaneously — 30s+ interval
• reactionNotifications: "off" on all agents (prevents event listener backlog)

Pod Operations

• Operator doesn't sync env changes to StatefulSet — must delete StatefulSet to force full reconcile. Resources (memory/cpu) DO sync.
• Image digest managed via kubectl patch only — manifests intentionally omit digest
• After CrashLoopBackOff, backoff reaches 5min — time cleanup to the brief running window
• v2026.3.28 needs 3Gi memory limit + --max-old-space-size=2048

Model IDs

• Opus 4.6: global.anthropic.claude-opus-4-6-v1 (has -v1)
• Sonnet 4.6: global.anthropic.claude-sonnet-4-6 (NO -v1)

ServiceAccount

• Never change pod SA — breaks Bedrock Pod Identity
• Use extraVolumes for separate kubectl SA token

Diagnostics

kubectl get pods -n opc-agents
kubectl logs <name>-0 -n opc-agents -c openclaw --tail=30
kubectl get events -n opc-agents --sort-by='.lastTimestamp'
kubectl top pod -n opc-agents
# Check all digests:
for a in admin devlead josh marketing milo; do
  echo "$a: $(kubectl get openclawinstance $a -n opc-agents -o jsonpath='{.spec.image.digest}')"
done