kk-deploy

name

kk-deploy

description

Build and deploy the KarmaCadabra Claude Agent SDK agents on Fargate. Use this skill whenever the user says "deploy", "build and push", "push to ECR", "redeploy", "rebuild Docker", or wants new code running on the KK swarm. Also use when discussing SDK agent Docker builds, ECR pushes, or the scheduled Fargate stack. Proactively use this after committing code changes that affect agent behavior (agents_sdk/, services/, lib/, scripts/kk/).

KK Deploy — Build + Push (SDK agent on Fargate)

The KK swarm runs in the terraform/sdk-agents/ stack, cluster karmacadabra-sdk-agents (us-east-1), in TWO modes:

• Scheduled buyers: EventBridge fires every 5 min → one short-lived task = one heartbeat. They pull :latest at task start, so a fresh push reaches them automatically within ~5 min.
• Persistent sellers (persistent = true in variables.tf — as of 2026-07-16: karma-hello, skill-extractor, voice-extractor, soul-extractor, validator; NEVER hardcode this list, discover it, see below): always-on ECS services (aws_ecs_service.seller, service.tf) with an in-container loop. They do NOT pick up a push on their own — after every image push run aws ecs update-service --cluster karmacadabra-sdk-agents --service kk-sdk-kk-<agent> --force-new-deployment --region us-east-1 for each. Deploys are STOP-THEN-START (min=0/max=100): a ~1-2 min presence gap per seller is expected and correct — the gap-free 100/200 rolling deploy made the new task cede to the old one via the singleton guard, so every seller deploy from 2026-07-08 to 2026-07-10 silently FAILED and rolled back (fixed 2026-07-10). Verify rollout: --query "services[].deployments[0].rolloutState" must reach COMPLETED, not FAILED.

There is no EC2, no SSH — the old OpenClaw EC2 pipeline was removed.

Architecture

• Docker image: Dockerfile.sdk-agent at repo root (Python + agents_sdk/, no shell/exec tools)
• ECR repo: 518898403364.dkr.ecr.us-east-1.amazonaws.com/karmacadabra/sdk-agent:latest
• Region: us-east-1
• Runtime: EventBridge Scheduler → ecs:RunTask on Fargate; task defs reference :latest
• Per-agent IAM: each task has its own role scoped to kk/<agent>-* + kk/anthropic-*
• Safe mode: agents run KK_EM_WRITES=0 (no real USDC spend) until flipped in variables.tf

Deploy a code change (agents_sdk/services/lib/scripts) — image only

Task definitions reference the :latest tag, so a fresh push is picked up by the next scheduled heartbeat automatically (buyers). No terraform needed for pure code changes. Persistent sellers additionally need the --force-new-deployment step below.

cd /mnt/z/ultravioleta/dao/karmakadabra

# 1. ECR login (token expires ~12h)
aws ecr get-login-password --region us-east-1 \
  | docker login --username AWS --password-stdin 518898403364.dkr.ecr.us-east-1.amazonaws.com

# 2. Build with --no-cache for code changes (cached layers serve stale code)
docker build --no-cache --platform linux/amd64 -f Dockerfile.sdk-agent -t karmacadabra/sdk-agent:latest .

# 2b. 🚨 MANDATORY — VERIFY THE IMAGE BEFORE PUSHING. Docker on Windows over the
# Z: drive can silently DROP freshly-written files from the build context (sync
# lag; --no-cache does NOT prevent it). A "successful" build (exit 0) can ship an
# image missing your newest .py -> ImportError crash-loop across the whole fleet
# in prod (2026-07-14 incident: concierge.py + secrets.py dropped, every agent
# died in build_context before reasoning; #Agents went silent). NEVER push an
# unverified image.
docker run --rm --entrypoint sh karmacadabra/sdk-agent:latest -c \
  'cd /app && python -c "from agents_sdk import secrets, runner, context, chatstate, convmem, tools; print(\"IMPORTS OK\")"'
# File parity: every agents_sdk/*.py on disk must be in the image (empty = good).
comm -23 <(ls agents_sdk/*.py | xargs -n1 basename | sort) \
  <(docker run --rm --entrypoint sh karmacadabra/sdk-agent:latest -c 'ls /app/agents_sdk/*.py' | xargs -n1 basename | sort)
# If IMPORTS fail or parity shows a missing file: RE-RUN the build (the files have
# settled now) and re-verify. Do NOT push until both checks are clean.

# 3. Tag + push to ECR
docker tag karmacadabra/sdk-agent:latest 518898403364.dkr.ecr.us-east-1.amazonaws.com/karmacadabra/sdk-agent:latest
docker push 518898403364.dkr.ecr.us-east-1.amazonaws.com/karmacadabra/sdk-agent:latest

# 4. PERSISTENT SELLERS ONLY — they never pick up a push on their own.
# 🚨 DISCOVER the list, never hardcode it. A hardcoded list of 4 silently skipped
# kk-validator for 2 days after it became persistent (2026-07-14): it kept serving
# an image >24h old while every deploy reported success. list-services cannot drift.
SVCS=$(aws ecs list-services --cluster karmacadabra-sdk-agents --region us-east-1 \
  --query 'serviceArns' --output text | tr '\t' '\n' | sed 's|.*/||')
echo "$SVCS"   # sanity-check: must match `persistent = true` in variables.tf
for svc in $SVCS; do
  aws ecs update-service --cluster karmacadabra-sdk-agents --service "$svc" \
    --force-new-deployment --region us-east-1 --query 'service.serviceName' --output text
done
# Wait for COMPLETED (FAILED = circuit breaker rolled back; check the new task's logs):
aws ecs describe-services --cluster karmacadabra-sdk-agents --services $SVCS \
  --region us-east-1 --query "services[].{name:serviceName,rollout:deployments[0].rolloutState}"

# 4b. PROVE the code actually landed — a COMPLETED rollout is not proof of freshness.
# Every service's running task must carry the SAME digest as ECR :latest.
LATEST=$(aws ecr describe-images --repository-name karmacadabra/sdk-agent --region us-east-1 \
  --image-ids imageTag=latest --query 'imageDetails[0].imageDigest' --output text)
for svc in $SVCS; do
  t=$(aws ecs list-tasks --cluster karmacadabra-sdk-agents --service-name "$svc" \
        --region us-east-1 --query 'taskArns[0]' --output text)
  d=$(aws ecs describe-tasks --cluster karmacadabra-sdk-agents --tasks "$t" \
        --region us-east-1 --query 'tasks[0].containers[0].imageDigest' --output text)
  [ "$d" = "$LATEST" ] && echo "OK    $svc" || echo "STALE $svc -> $d"
done

For a config-only change (SOUL.md / HEARTBEAT.md), a cached build is fine — drop --no-cache.

Deploy an infra change (new agent, IAM, schedule) — terraform

cd /mnt/z/ultravioleta/dao/karmakadabra/terraform/sdk-agents
terraform plan -input=false      # review
terraform apply -input=false -auto-approve

New task-definition revisions and per-agent IAM roles are created; EventBridge targets update to the new revision automatically.

Verify (don't wait 5 min — trigger one task)

⚠️ The --overrides and --started-by flags are MANDATORY (2026-07-08 incident: a bare run-task against a seller family inherits KK_LOOP_SECONDS=90 and becomes an IMMORTAL duplicate task that spends real USDC and collides its IRC nick into <agent>_ / <agent>__ variants — it never dies on its own and must be hunted down with stop-task). The overrides force: one-shot (KK_LOOP_SECONDS=0), no spend (KK_EM_WRITES=0), no IRC connection (KK_IRC_MODE=off, so no nick collision), and skip the singleton guard (KK_SINGLETON_CHECK=0, so it doesn't cede to the live service task and exit before testing anything).

cd /mnt/z/ultravioleta/dao/karmakadabra/terraform/sdk-agents
SUBNET=$(terraform output -json subnets | python3 -c "import json,sys;print(json.load(sys.stdin)[0])")
SG=$(terraform output -raw security_group_id)
aws ecs run-task --cluster karmacadabra-sdk-agents --launch-type FARGATE --region us-east-1 \
  --task-definition karmacadabra-sdk-kk-karma-hello \
  --started-by "manual-verify-$(whoami)" \
  --overrides '{"containerOverrides":[{"name":"kk-karma-hello","environment":[
    {"name":"KK_LOOP_SECONDS","value":"0"},
    {"name":"KK_EM_WRITES","value":"0"},
    {"name":"KK_SINGLETON_CHECK","value":"0"},
    {"name":"KK_IRC_MODE","value":"off"}]}]}' \
  --network-configuration "awsvpcConfiguration={subnets=[$SUBNET],securityGroups=[$SG],assignPublicIp=ENABLED}" \
  --query 'tasks[0].taskArn' --output text

(To verify the IRC join path specifically, deploy and watch the service task's logs instead of launching a second live instance — two live connections per identity is the exact bug this repo just fixed.)

Then read the logs (stream = ecs/<agent>/<task-id>):

aws logs get-log-events \
  --log-group-name /ecs/karmacadabra-sdk-agents/kk-karma-hello \
  --log-stream-name "ecs/kk-karma-hello/<TASK_ID>" \
  --region us-east-1 --query 'events[*].message' --output text

Look for:

• heartbeat start: agent=... em_writes=False — boot OK, safe mode
• heartbeat done — full success
• A 400 credit balance is too low means the kk/anthropic account needs funding, not a code bug.
• IRC checks (Connected as kk-... + Joined #karmakadabra) live in the service/scheduled task logs, not in manual verify runs (which run with KK_IRC_MODE=off, see above).
• Nick in use in any log = duplicate live instance → check aws ecs list-tasks for rogue tasks (the kk-irc-nick-collision CloudWatch alarm fires on this too).

Common Issues

ECR push fails with "no basic auth credentials"

Re-run the ECR login (tokens expire after ~12h).

Task exits with code 1 right after boot

Usually the Anthropic API: credit balance is too low. Fund the account behind kk/anthropic. Otherwise check the traceback in CloudWatch — boot fetches secrets via the per-agent task role.

Old code still running after push

Docker cache served a stale image. Always --no-cache for code changes; confirm the new digest:

aws ecr describe-images --repository-name karmacadabra/sdk-agent --region us-east-1 \
  --query 'sort_by(imageDetails,&imagePushedAt)[-1].[imagePushedAt,imageDigest]' --output text

When to Deploy

• After changing agents_sdk/, services/*.py, lib/*.py, scripts/kk/*.py — image push
• After changing openclaw/agents/*/SOUL.md or HEARTBEAT.md — image push (prompts are baked in)
• After changing terraform/sdk-agents/*.tf (new agent, IAM, schedule) — terraform apply
• NOT needed for: docs, contracts, the old agents/* FastAPI sellers (not deployed)