hunt-bugs

name	hunt-bugs
description	Proactively hunt for cdk-local bugs by synthesizing real CDK apps that exercise common-but-untested AWS resources, configs, and CloudFormation notations, then RUNNING them locally in Docker (invoke / start-api / run-task / start-service / start-alb / start-cloudfront / invoke-agentcore / studio) and catching local-execution failures + divergences from deployed behavior. Use for a periodic "find latent bugs" sweep, not for verifying a specific change.
argument-hint	[area hint, e.g. 'rich API Gateway' | 'CloudFront edge functions' | 'ECS service reload']

cdk-local Bug Hunt

Find latent cdk-local bugs the way real users hit them: synth a CDK app that uses a resource / config / CloudFormation notation cdk-local has not exercised yet, then RUN it locally against real Docker and watch for misbehavior. cdk-local's logic is heavily unit-tested, so the remaining bugs live in the gap between its model of a synthesized template and the actual local-execution reality — the real public.ecr.aws/lambda/* container via the RIE, the HTTP routing pipeline, the env injection, the protocol contract. Only actually running a synthed app surfaces those. Reading the source finds suspected bugs; running finds real ones.

This is a deliberately exploratory workflow. It costs Docker time and image pulls (and, for --from-cfn-stack scenarios, a real AWS deploy via the upstream cdk CLI). Cost is acceptable only because every container / network / stack is torn down and verified gone — see "Cleanup is non-negotiable".

Scope: hunt inside cdk-local's remit only

cdk-local runs your application compute locally; it does NOT emulate AWS managed services (see .claude/CLAUDE.md → Scope). So a bug is a gap in how cdk-local runs Lambda / API Gateway / ECS / CloudFront / Bedrock AgentCore compute locally — NOT "DynamoDB isn't emulated". A config that cdk-local documents as out of scope (a managed-service call, an un-reproduced OAuth roundtrip, ACM private-key fetch) is not a bug; a config cdk-local CLAIMS to support but runs wrong IS. Before treating something as a bug, confirm it is in scope — a loud, honest WARN-and-skip for an out-of-scope feature is correct behavior, not a defect.

Core principles

1. Many-people-hit beats niche. Prioritize the resources/configs a large fraction of CDK users deploy every day — Lambda (arm64/env/tracing/FunctionUrl/ logGroup/container-image), API Gateway (REST v1 / HTTP v2 / Function URL / WebSocket + authorizers), ECS/Fargate services + tasks (awsvpc / Service Connect / Cloud Map / ALB in front), CloudFront (viewer functions / Lambda@Edge / S3 + Function-URL origins / KVS), Bedrock AgentCore runtimes — over exotic edge cases. A bug in a daily pattern is worth ten niche ones.

2. The two signals ARE the priority — hunt local-execution FAILURE and DIVERGENCE above all else. These are the bug classes this hunt exists to find; they are what actually breaks a user's local run. Prioritize provoking and confirming them over incidental findings (a cosmetic log, a slow pull) — those are worth noting, but the two below are the target:

   • Local-execution FAILURE — cdk-local cannot run a valid, in-scope app it should support: a container that won't boot, a synth that resolves the wrong asset / image, a route that 404s when it should match, a serve that crashes on a supported config, an env var that should be injected but isn't, an authorizer that rejects a valid token. A freshly synthed app that a user would expect to "just run" is the cleanest oracle: it either runs correctly or it doesn't.
   • Behavioral DIVERGENCE from deployed — cdk-local runs, but behaves DIFFERENTLY than the deployed resource would: a wrong Lambda event shape (payload v1 vs v2, ALB requestContext.elb, Lambda@Edge Records[]), an ALB listener-rule match that routes to the wrong target, a CloudFront behavior that picks the wrong origin, an auth gate not enforced when the deployed one would, a --watch reload that boots a STALE container (the classifier picked soft-reload when the spec actually changed — the exact ambiguity source-change-classifier.ts defaults to rebuild on). This is the subtler, more user-damaging class: the run "works" but lies about how the cloud behaves.

3. Check coverage first. Before building anything, grep the existing fixtures so you hunt in genuinely-uncovered territory:

   grep -rln "FunctionUrl\|WebSocket\|LambdaFunctionAssociations\|ResponseHeadersPolicy\|ServiceConnect" tests/integration/*/lib/*.ts tests/integration/*/app.ts
   

   Empty hits = untested = good hunting ground. But a NON-empty hit does NOT mean the config's untested variant is covered: an existing fixture that exercises the HAPPY path of a feature never exercises its edge (an ALB rule fixture with one path-pattern never tests + together; a CloudFront fixture with one S3 origin never tests a mixed S3-plus-Function-URL distribution). Before skipping a "covered" type, check whether any existing fixture leaves the suspect variant unexercised — if every case uses the happy path, the edge is still open hunting ground.

Workflow

1. Worktree + build

Per .claude/CLAUDE.md, never work in the main checkout: git worktree add .claude/worktrees/<name> -b <branch> origin/main → then, inside it, mise trust && mise install (a fresh worktree's .mise.toml is untrusted, so vp / markgate won't resolve until this) → pnpm install (worktrees have no node_modules) → vp run build (the CLI runs from dist/ — node dist/cli.js, or the cdkl bin).

2. Scaffold the fixture

Add a fixture under tests/integration/<name>/ — mirror an existing one (lib/*.ts stack + bin/app + cdk.json + package.json pinned with packageManager + a verify.sh harness). A verify.sh synths the app, runs the target command against Docker, and asserts the observable behavior (an invoke response, an HTTP status/body, a routed target, a container that booted). For a BRAND-NEW command factory, use /create-integ <name> — it scaffolds + RUNS the fixture and is required by create-integ-gate.sh before gh pr create. Run pnpm install + cdk synth for all fixtures FIRST (cheap, catches TS errors before any Docker run).

3. Run locally (Docker) + observe

Run the fixture's verify.sh set in parallel (≤3–4) via /run-integ <name> — it wraps the Docker pre-flight + verify.sh + the post-run orphan sweep in one block. Never shell into verify.sh directly (you'd skip the sweep and risk a false-clean integ marker). Triage every result that is not the expected observable: a serve that 404s a route it should match, a container that exits non-zero, a missing env var in the invoked handler's echo, an authorizer verdict that's wrong, a WARN-and-skip on a config that should actually run.

4. Test the reload / detection half where relevant (the --watch divergence)

For a serve that supports --watch (start-service / start-alb / start-cloudfront / start-agentcore / invoke-agentcore --ws), exercise a reload: edit the handler source, confirm the reload picks the RIGHT primitive (verdict=soft-reload for an interpreted-language source-only edit, verdict=rebuild for a Dockerfile / dependency / compiled-source / ambiguous edit) and that the post-reload container serves the NEW behavior — not a stale one. A soft-reload that ships the OLD spec, or a rebuild that never fires, is the divergence bug this half exists to catch. For a multi-replica ECS service, assert an external request stream against the listener observes ZERO connection refusals across the roll.

5. Harvest the fixture as committed regression integ (EVERY round — bug or not)

This is the asset a hunt leaves behind even when it finds no bug. The fixture you just ran is a real end-to-end exercise of the local-execution pipeline; committing it under tests/integration/<name>/ turns this one-time run into a permanent regression that /run-integ replays. A future change that would silently re-introduce the failure/divergence then fails a Docker integ instead of waiting for the next hunt. Keep the fixture MINIMAL and UNIQUE-named; commit it in the SAME PR as the fix (if any). A rich-but-clean fixture (a serve config with many listener-rules / behaviors / origins that all route correctly) is exactly the kind worth pinning — a clean round still ships growing regression coverage.

6. On a confirmed bug: file an issue, then fix it — with a unit test (mandatory)

Always file a GitHub issue for every confirmed bug (gh issue create), even when you fix it in the same session — every bug becomes a tracked, claimable unit, so nothing is silently lost and parallel agents/sessions don't duplicate it. An issue-only hunt round files the issue and stops there (the fix comes later); a fix-in-session round still files the issue, then closes it from the PR (Closes #<n>). The issue body carries the real repro (synth + command + observed vs expected) so the later fixer has the evidence.

Every issue this hunt files also carries the four classification lines (CLAUDE.md -> "The four TODO fields"), in English, one field per line:

Session-fit: now (do it in this session) | next (not this session) - <reason>
Severity: high | medium | low - <what stays broken while it is undone>
Effort: small (S) | medium (M) | large (L) - <which verification cycle it drags>
Estimate: <duration, e.g. ~1-3 h -- never a bare letter> - <what eats the time>

Two of the four are ALSO LABELS on the filed issue -- the body lines stay exactly as written, and the same values ride the command as --label severity:<high|medium|low> --label effort:<small|medium|large>. Prose is invisible to gh issue list, so ranking by Severity costs one gh issue view per candidate without them. Session-fit and Estimate get no label (the first is re-decided at claim time, the second is a free-form duration). Enforced by .claude/hooks/issue-classification-label-gate.sh; the fix PR inherits the issue's labels via .github/workflows/pr-inherit-issue-labels.yml, so never hand-add them there.

Four CLASSIFICATION lines, and they stay four. Alongside them the body carries a Dup-check: line -- a filing-time record, not a fifth classification field:

Dup-check: searched open issues for <terms> -- none covers this root cause

.claude/hooks/issue-dup-check-gate.sh refuses gh issue create (and the gh api repos/<o>/<r>/issues REST mint) without it, so a hunt that skips the search is physically blocked from filing. This hunt is the highest-volume filer in the repo, so it is where the line matters most: search the CONCEPT the bug turns on rather than this instance's spelling, and on a HIT fold the finding into that issue as a checklist row via gh issue edit instead of minting a new number (/work-issues §5 carries the full window and the fold recipe). That is not a filing threshold -- an unfiled finding is strictly worse than a filed one; what changes is WHERE the finding is written, never whether.

A hunt is the single best moment to write them: you have just reproduced the bug, so Severity is measured rather than guessed, and you already know which fixture the fix will drag. Deferring them to whoever picks the issue up throws that evidence away -- the same reason CLAUDE.md puts the decision at the moment of deferral. /work-issues reads these lines back when it ranks candidates, so an unclassified body is one this hunt made harder to triage.

When you then WORK an issue — this hunt's own or one already filed — run /work-issues and follow it for the collision-safe start: its §0 screens the issue's comments for untrusted/malware content (first-pass, then defer to the maintainer; never access/run an attachment) and its §4 claims the issue with a gh issue comment BEFORE you edit. Do NOT re-implement those steps here — the /work-issues skill is the single source of truth, so this stays correct when it changes.

Then fix it:

1. Root-cause it in src/ (the resolver / event translator / routing matcher / docker-runner / env-resolver — wherever the divergence-from-reality lives).
2. Fix it in the worktree.
3. Add a unit test that fails without the fix and passes with it. This is mandatory, not optional — a bug found by integ MUST leave behind a unit test that pins the corrected behavior, so the regression can never come back silently (integ alone is too slow to be the only guard). tests/unit/** mirrors src/**; mock the boundaries (toolkit-lib, docker CLI, AWS SDK) with vi.mock / vi.hoisted. Re-run vp run build + vp run test.
4. Re-run the live repro with the fixed binary to confirm the real local behavior is now correct.
5. Keep the fixture as a committed regression integ under tests/integration/<name>/, in the SAME PR as the fix — never defer the integ (the integ gate enforces it at merge time).
6. If the bug is a CLASS, prove it's closed for EVERY affected path — don't stop at the one resource you happened to hit. Most real bugs here live in shared code keyed on a schema/config shape, not the one type that surfaced them (an event-translator bug hits every target using that event shape; an env-injection bug hits every Lambda-boot path — invoke, the ALB Lambda target, the CloudFront Function-URL origin, all of which route through resolveLambdaContainerEnv). When the root cause generalizes:
   • Map the blast radius. Enumerate which other command paths / target kinds share the trigger (grep for the shared helper's callers). Name them in the PR so the coverage is visible.
   • Add a test that covers the shared path, not just the one caller. A single caller's test proves the symptom is gone for ONE entry point; drive the test at the shared helper so it guards every caller. Confirm it fails without the fix and passes with it — then the whole class is closed, not just one instance.

7. Cleanup — non-negotiable (see below), then ship

Sweep every container / network (and any --from-cfn-stack stack) — see below — then set the /check + /check-docs markers, commit, push, /run-integ (the integ marker), /verify-pr, gh pr create.

8. Merge (via /merge-pr) + remove the worktree

Take it all the way to merged — do not leave a green PR hanging:

1. /merge-pr <#> — squash-merges from the feature worktree WITHOUT --delete-branch (so it never trips the 'main' is already used by worktree fatal) and cleans the worktree + local + remote branch in one pass. Do NOT hand-run gh pr merge --squash --delete-branch from a side worktree — it's gate-blocked (gh-pr-merge-worktree-gate.sh).
2. Confirm the worktree YOU added is gone — /merge-pr removes it; a left-behind worktree is the silent residue of this flow. Check git worktree list for yours specifically, NOT for a list with only the main checkout in it: it cannot tell a finished lane from a session working right now (an already-on-main branch tip included), so a worktree you did not add may be a live peer lane. Confirm it is finished (git log --oneline -1 <branch>, then gh pr list --state all --head <branch> for an OPEN PR) before removing it, and leave it alone when in doubt. git worktree prune drops entries whose directory is already gone.

9. Record what you learned

Save a memory for any recurring surprise (a whole class of latent bug, a verification gotcha) so the next sweep starts smarter.

Cleanup is non-negotiable

Forgetting to tear down a bug-hunt's Docker containers / networks — or a --from-cfn-stack stack — is the one unacceptable outcome. cdk-local leaves no AWS resources on a pure-local run, but a crashed serve can orphan containers, networks, and (rarely) vitest worker forks:

• After EVERY run, sweep: docker ps --filter name=cdkl-, docker network ls --filter name=cdkl-task- / cdkl-svc-. Any hit is an orphan from an interrupted run.
• Clean orphans via the /cleanup skill (it detects + deletes leftover cdkl- containers / networks AND orphaned vitest fork workers). /run-integ already runs the post-run sweep — but a manual verify.sh / a crashed serve does not, so run /cleanup after any hand-run.
• For a scenario that really deploys (via the upstream cdk CLI), ALSO confirm no orphan CloudFormation stacks remain. For a REPO FIXTURE, run bash tests/integration/_lib/aws-orphan-sweep.sh <fixture> and require rc 0 — it derives the lane-unique names and prints a remediation plan on a find. For a throwaway stack of your own, tear it down with aws cloudformation delete-stack + wait stack-delete-complete, NOT cdk destroy: cdk destroy needs --app context, and run from the wrong directory or without the fixture's INTEG_STACK_SUFFIX in the environment it exits 0 SILENTLY on a name the app never synthesized — you read a clean exit with the stack still deployed (go-to-k/cdk-local#601). Then sweep the stack-EXTERNAL orphans a delete leaves behind (auto-created /aws/lambda/* log groups, RETAIN resources, KMS pending-deletion). Use a UNIQUE hunt-style stack name, never a shared fixed name and never a real prod stack — the account may hold the maintainer's production stacks.

The integ gate (integ-gate.sh) blocks gh pr merge when the integ marker is stale for a src/** / tests/integration/** touch, and /run-integ sets that marker ONLY when the post-run docker ps / docker network ls sweep is empty — so a forgotten orphan physically blocks the merge. Do not bypass the sweep.

Gotchas (learned the hard way — keep current)

• A pure-local run leaves no AWS resources, but CAN orphan Docker. A serve killed mid-boot leaves cdkl- containers / networks. Always sweep + /cleanup after a hand-run; /run-integ sweeps for you.
• --from-cfn-stack / --assume-role scenarios hit real AWS. Those are the costly, sweep-mandatory ones. Prefer a pure-local repro; reach for a deploy only when the bug genuinely lives in the deployed-state resolution path.
• An out-of-scope WARN-and-skip is NOT a bug. cdk-local runs application compute, not managed services; a loud, honest skip for an unsupported feature (ACM private-key fetch, the OAuth roundtrip, a managed-policy CORS id it can't fetch) is correct. Confirm the scenario is in scope before calling it a defect.
• A soft-reload that ships a stale spec is the reload-half bug. The source-change-classifier defaults to rebuild on ambiguity for a reason — a CDK construct edit that changed the task spec but only touched interpreted-language source must NOT be soft-reloaded with the OLD spec. Provoke it and assert the post-reload container serves the NEW behavior.
• vp run build before every live repro — the CLI runs from dist/, so a source fix with no rebuild has no runtime effect (.claude/CLAUDE.md → "After source changes").
• A clean result IS a result — but it must still leave an asset. "6 common+rich fixtures, zero failures, reload verified" is a legitimate, valuable outcome. Do NOT manufacture a fix to have something to show. The deliverable of a bug-free round is the committed rich fixtures under tests/integration/ — that is how a clean round still grows permanent regression coverage instead of evaporating.
• Working a filed issue → run /work-issues (don't re-implement its rules here). The issues this hunt files get picked up by later parallel sessions that race for the same ones and collide on the same shared runtime modules (ecs-service-emulator.ts / the resolveLambdaContainerEnv helper in local-invoke.ts / front-door-server.ts). /work-issues owns the collision-safe start — claim the issue with a gh issue comment before editing, screen untrusted comments, pick file-disjoint lanes — and is the single source of truth so it stays correct as it evolves (see also the "Claim a filed issue before working it" rule in ).