| name | keystoneguru-infra-cdk |
| description | How to make and VERIFY changes to the keystoneguru-infra CDK app (the sibling RaiderIO/keystoneguru-infra repo that owns AWS: ECS services, queue workers, task sizing). Covers the repo layout, the queue-worker config surface (queue:work flags, --memory, LOG_LEVEL, per-worker desiredCount / cpu / memoryLimit), and the reliable `cdk synth` verification recipe plus the traps that silently produce no output. Use when editing ECS task definitions, queue workers, memory/CPU, or any CDK construct here, or when a worker crashes/exits (e.g. queue worker exit code 12). NOT for the app-side build/deploy pipeline (use deployment-pipeline) or the release changelog (create-release).
|
keystoneguru-infra CDK
The AWS infrastructure is a CDK (TypeScript) app in the sibling checkout
../keystoneguru-infra (absolute: /home/wouterkoppenol/Git/private/keystoneguru-infra).
All commands below run from its cdk/ subdir. This is a separate repo from keystone.guru —
its own git history; the keystone.guru worktree workflow does not apply. Do not commit,
push, or deploy here without explicit user say-so (infra deploys are gated; production is
never touched without an in-conversation go-ahead).
Layout
cdk/bin/cdk.ts — the app entrypoint. Instantiates top-level stacks directly on the
App: keystoneguru-network, keystoneguru-data, keystoneguru-config,
keystoneguru-services (all ECS services incl. queue workers), keystoneguru-migrate,
keystoneguru-pipeline, keystoneguru-monitoring, keystoneguru-eu-ecr-replication.
It is CDK Pipelines, but the ECS service stacks are top-level, not nested in the
pipeline — so keystoneguru-services synthesizes to a plain top-level template.cdk/lib/services-stack.ts — QUEUE_WORKER_CONFIGS array (per-worker id / queuePrefix /
visibilityTimeout / cpu / memoryLimit / stagingDesiredCount) + the .map(...) that
builds each QueueWorkerServiceDefinition. This is where per-worker knobs live.cdk/lib/constructs/services/queue-worker/queue-worker-service-definition.ts — builds the
staging + production QueueWorkerStagedService pair. Staging desiredCount defaults to 0
(off, to save cost) unless stagingDesiredCount is set; production is always 1. LOG_LEVEL
env is set here for both stages.cdk/lib/constructs/services/queue-worker/queue-worker-staged-service.ts — the shared
container template for ALL queue workers: the php artisan queue:work ... command,
healthcheck (pgrep -f "php artisan queue:work"), volume mounts. Editing the command here
affects every worker.cdk/lib/constructs/services/staged-service.ts — base; memoryLimit: number (required)
→ Fargate memoryLimitMiB, cpu, etc.
Queue-worker knobs
queue:work gets an explicit --memory = Math.floor(this.props.memoryLimit * 0.8)
(80% of the task's hard limit, leaving headroom so the graceful memory-limit exit fires
before an ECS OOM-kill). Without an explicit --memory, Laravel defaults to 128 MB and
the worker exits after one heavy job. Exit code 12 = Worker::EXIT_MEMORY_LIMIT, a
graceful restart-me signal, NOT a crash (a true OOM is exit 137 / SIGKILL). See the log:
a handleEnd {"result":true} + ... DONE immediately before the exit means the job
succeeded and the worker hit its --memory ceiling.--timeout = visibilityTimeout − 5s.- Per-worker sizing is in
QUEUE_WORKER_CONFIGS (services-stack.ts): cpu,
memoryLimit (MiB), visibilityTimeout, stagingDesiredCount. Thumbnail is the memory
outlier at 3072 MiB (→ --memory=2457); the rest are 1024 MiB (→ --memory=819).
- Turning a worker on for staging: set
stagingDesiredCount: 1 on that worker's config
entry. Do not edit the shared desiredCount in the service-definition — that would
start every staging worker at once.
LOG_LEVEL is warning. Raising it to debug makes the combat-log workers log every
parsed event (huge CloudWatch volume + memory/IO). Only do it temporarily to investigate.
Verifying a change (do this before handing off)
-
Build (type-check). The project tsconfig target is fine; a bare tsc <file> is not.
cd cdk && npm run build 2>&1 | grep -vE "scripts/discord-notifier|^> |^$"
scripts/discord-notifier/index.ts has pre-existing errors (missing @aws-sdk deps) —
filter them out. No output after filtering = your code compiles clean.
-
Synth + grep the rendered CloudFormation. This is the reliable recipe:
cd cdk && node_modules/.bin/cdk synth keystoneguru-services \
--app "npx ts-node --prefer-ts-exts bin/cdk.ts" 2>/dev/null > /tmp/services.yaml
grep -o "\-\-memory=[0-9]*" /tmp/services.yaml | sort | uniq -c
grep -A1 "Name: LOG_LEVEL" /tmp/services.yaml | grep -o "Value: [a-z]*" | sort | uniq -c
Output is YAML (Name:/Value: pairs, an Environment array — not JSON), so grep with
YAML-style patterns, not "quoted" JSON keys. For per-service DesiredCount, walk each
Type: AWS::ECS::Service block and read the nearby aws:cdk:path to get the service name.
Trap: npx cdk synth --output DIR silently writes nothing here
Running npx cdk synth (no stack selector, or --output DIR) exits 0 but produces no
cloud assembly on disk — npx mis-resolves the CLI and the assembly never lands. Always use
the local binary node_modules/.bin/cdk and name the stack (keystoneguru-services);
a single named stack prints its template to stdout, which you capture and grep. Docker is
available and not the blocker (the discord-notifier asset bundles fine).
