run-integ

name	run-integ
description	Run integration tests (deploy + destroy) against real AWS. Use when you need to verify cdkd works end-to-end with actual AWS resources.
argument-hint	<test-name|all> [--synth-only] [--no-destroy]

Integration Test Runner

Run integration tests against a real AWS account. These tests deploy actual AWS resources, verify them, and clean up.

Arguments

• test-name: Which test to run. Run ls tests/integration/ to see all available tests. If not specified, use the AskUserQuestion tool to ask which test to run, showing the available options.
• all: Run all tests
• --synth-only: Only run synthesis, skip deploy/destroy
• --no-destroy: Deploy but don't destroy (for debugging)
• --deploy-args "<args>": Forward extra arguments to the cdkd deploy invocation (e.g. --deploy-args "--aggressive-vpc-parallel"). Use when validating an opt-in deploy flag end-to-end against bench-cdk-sample etc. — the destroy step is unaffected (opt-in flags so far are deploy-only).

Steps

1. Build first: Run vp run build to ensure dist/ is up to date.

2. List available tests: Run ls tests/integration/ to discover all test directories dynamically. Do NOT rely on a hardcoded list.

3. Determine state bucket: Resolve dynamically via aws sts get-caller-identity --query Account --output text to get the account ID, then construct cdkd-state-{accountId} (region-free, the current default since PR #62 / v0.11.0). If that bucket doesn't exist, fall back to the legacy cdkd-state-{accountId}-us-east-1 and note the deprecation in the report.

4. Pre-flight orphan scan (mandatory — fail fast on prior-run leftovers instead of going through CREATE + rollback):

   Before invoking deploy, scan AWS for resources matching the stack name from this test that have no business existing yet. The scenario this catches: a previous integration run was killed mid-deploy, leaving orphan Event Source Mappings / Lambda functions / ENIs / IAM roles whose names match the stack about to be deployed. cdkd's diff calculation does NOT see these (they're not in state), so the deploy attempts CREATE — which collides with the orphans, fails immediately with ResourceAlreadyExists, and forces a CREATE-then-rollback cycle. Failing at the start is much cheaper than partway through.

   Synth first (without deploy) to learn the stack name and the resource types in the template, then for each scenario in scope run a targeted scan:

   # Always run these (cheap, broadly applicable):
   aws s3 ls s3://<bucket>/cdkd/<StackName>/ --region us-east-1
   aws iam list-roles --query 'Roles[?contains(RoleName, `<StackName>`)].RoleName' --output text
   aws lambda list-functions --region us-east-1 \
     --query 'Functions[?contains(FunctionName, `<StackName>`)].FunctionName' --output text
   
   # Run when the template uses Lambda EventSourceMapping (the orphan ESM
   # case bit cdkd in 2026-05-02: AlreadyExists + rollback cycle on a fresh deploy):
   aws lambda list-event-source-mappings --region us-east-1 \
     --query 'EventSourceMappings[?contains(FunctionArn, `<StackName>`)].[UUID,FunctionArn]' --output text
   
   # Run when the template uses VPC + Lambda VpcConfig (hyperplane ENIs
   # outlive their function):
   aws ec2 describe-network-interfaces --region us-east-1 \
     --filters "Name=description,Values=AWS Lambda VPC ENI-<StackName>*" \
     --query 'NetworkInterfaces[].[NetworkInterfaceId,Status]' --output text
   

   If anything is found, abort the test run with a clear report listing the orphans and the cleanup commands the user should run (aws lambda delete-event-source-mapping --uuid …, cdkd state destroy <StackName> --yes, etc.) — do NOT proceed with deploy. Resuming on top of orphans is the failure mode this step exists to prevent.

   If nothing is found, the deploy can proceed cleanly.

5. Run the test(s):

   Dispatch: a verify.sh in tests/integration/<test-name>/ is for tests with non-standard flows (drift-injection, multi-stage validation, etc.) — the script owns its own deploy + verify + destroy cycle. The standard flow below is for plain "deploy this stack and destroy it" smoke tests. Pre-flight (step 4) and the post-run verification (steps 6 + 7) apply to BOTH paths — they are the safety net that catches a buggy verify.sh leaking resources.

   CHECK FOR verify.sh BEFORE PICKING THE FIXTURE — the standard-flow branch is effectively unreachable from an agent session. The branch tells you to invoke node ../../../dist/cli.js deploy … directly, and the harness's auto-approval classifier refuses a direct cdkd deploy, so a fixture WITHOUT a verify.sh dead-ends after the dispatch. Measured 2026-08-20 on bench-cdk-sample (no verify.sh): bash verify.sh returned rc=127 in 0 seconds, and the standard-flow fallback was then denied. rc=127 means the file does not exist, and bash says so on STDERR — bash: verify.sh: No such file or directory, verified on both bash 5.x and macOS system bash 3.2. If that line is missing from what you are reading, you redirected stderr to a log and are reading the tail; go look at it rather than interpreting the exit code alone. (This paragraph previously claimed the failure was silent. It is not — the report it came from had redirected the output.) The practical workaround is a SELECTION decision, made here rather than after the dead-end: when the goal is a marker (integ-broad / integ-destroy), pick a broad-set fixture that HAS a verify.sh (ls tests/integration/<name>/verify.sh before committing to the name). Only run the standard flow when a human is driving the shell and can approve the deploy / destroy invocations.

   • Navigate to tests/integration/<test-name>/
   • Ensure dependencies: npm install if node_modules doesn't exist
   • If tests/integration/<test-name>/verify.sh exists, run it instead of the standard flow:
     • AWS_REGION=us-east-1 STATE_BUCKET=<bucket> bash verify.sh
     • The script is responsible for its own deploy + destroy cycle. Steps 6 (verify cleanup) and 7 (auto-cleanup orphans) STILL run after — do not skip them.

6. Verify cleanup:

   • Check aws s3 ls s3://<bucket>/cdkd/ --region us-east-1 to confirm no leftover state
   • The state bucket is VERSIONED (cdkd bootstrap enables it), so that listing shows nothing while every prior version is still readable: aws s3 rm writes a DELETE MARKER, it does not remove content. For any fixture that WRITES a secret into state — the redaction / scrub / drift fixtures do it deliberately, to simulate a pre-GHSA binary — "the object is gone" is not "the content is gone", and the difference is a disclosure. Check versions, not just the current object:

     # Per state/lock key the fixture touched. Non-empty = content still readable.
     aws s3api list-object-versions --bucket <bucket> --prefix "cdkd/<Stack>/<region>/state.json" \
       --query "([Versions, DeleteMarkers][])[?Key=='cdkd/<Stack>/<region>/state.json'].VersionId" \
       --output text
     

     Two bugs of this shape shipped on 2026-08-19 and BOTH survived a green run: a fixture's version sweep lived only in cleanup, which runs from the trap the success path DISARMS (so on the normal path it never ran — one key had accumulated 30 versions), and the sweep itself iterated with printf '%s' | tr | while read, which drops the LAST field because there is no trailing newline. Neither is visible from reading the script; both are obvious the moment you COUNT what S3 holds afterwards. If a fixture seeded a secret, grep the surviving versions for it rather than assuming the sweep worked.
   • Also verify actual AWS resources are gone by checking with stack name prefix filters. Get stack names from the synth output, then for each stack name query AWS APIs filtered by that prefix:
     • aws iam list-roles --query 'Roles[?contains(RoleName, \{StackName}`)].RoleName'`
     • aws lambda list-functions --region us-east-1 --query 'Functions[?contains(FunctionName, \{StackName}`)].FunctionName'`
     • aws s3api list-buckets --query 'Buckets[?contains(Name, \{stackName-lowercase}`)].Name'`
     • aws ecr describe-repositories --region us-east-1 --query 'repositories[?contains(repositoryName, \{stackName-lowercase}`)].repositoryName'`
     • aws dynamodb list-tables --region us-east-1 --query 'TableNames[?contains(@, \{StackName}`)]'`
     • For VPC-based tests also check: aws ec2 describe-vpcs --filters "Name=tag:Name,Values={StackName}/Vpc" ...
     • For FSx-based tests also check (issue #1113): destroy keeps CFn parity, so runs with API defaults, which TAKE a chargeable final backup for Windows/ONTAP (observed on OpenZFS too; does not prevent it). The backup usually carries NO tags ( defaults to false), so a tag/name scan reports clean over a live billing backup; attribute by the backup's persisted instead: {fs-id}`].[BackupId,Lifecycle]'aws fsx describe-backups --region us-east-1 --query 'Backups[].{Id:BackupId,FsId:FileSystem.FileSystemId,Type:FileSystem.FileSystemType,Created:CreationTime}'aws fsx delete-backup --backup-id {id} --region us-east-1`.

7. Auto-cleanup orphans (mandatory when destroy didn't fully succeed):

   Trigger this step whenever any of the following is true:

   • The destroy step in step 5 reported a non-zero error count (e.g. "X failed to delete")
   • Step 6 found a leftover S3 state file
   • Step 6 found any AWS resource matching the stack name prefix

   What to do:

   • For VPC-attached Lambda failures (the most common pattern), the typical orphan set is, in delete order:
     1. Lambda hyperplane ENIs (aws ec2 describe-network-interfaces --filters "Name=vpc-id,Values=<vpc>" → aws ec2 delete-network-interface). Some may be in-use initially — re-poll until they go available, then delete.
     2. SecurityGroups (aws ec2 delete-security-group) — must come after the ENIs that reference them are gone.
     3. Subnets (aws ec2 delete-subnet) — must come after every ENI in them is gone.
     4. VPC (aws ec2 delete-vpc) — last.
   • For S3 state orphans: aws s3 rm s3://<bucket>/cdkd/<StackName>/ --recursive. (cdkd state orphan <StackName> is the cdkd-native equivalent and also handles the lock key.)
   • For other resource types, infer the right delete order from CloudFormation dependency rules (children before parents).
   • Always specify the correct region (--region).
   • Re-run step 6 after cleanup to confirm zero orphans remain.

   Never end the run with orphan resources still present in AWS. Cost (NAT GW alone is ~$1/hr) and account hygiene make this non-negotiable. If a resource genuinely cannot be deleted after reasonable retries, surface it to the user with the exact ID, region, and what was tried — but only after the auto-cleanup pass.

8. Report results: Show pass/fail for each test, including resource counts and timing. Always state explicitly "destroy completed: 0 errors, 0 orphans" or itemize what remained / what was force-cleaned.

9. Set the integ-destroy markgate marker (only on full clean success):

   When — and ONLY when — all of the following hold:

   • the destroy step finished with 0 errors,
   • step 6 found 0 leftover resources,
   • step 7 was either skipped (because nothing to clean up) or completed with the post-cleanup re-check showing 0 orphans,

   record the gate so subsequent gh pr merge calls are unblocked:

   mise exec -- markgate set integ-destroy || {
     echo "markgate set integ-destroy FAILED — the marker was NOT recorded." >&2
     exit 1
   }
   

   Check the exit code; do not fire and forget. Under the old hash: files mode this command could not fail. The gate now runs markgate 0.4's hash: diff (see .markgate.yml), where set exits 2 if origin/main is unresolvable in this worktree or the branch has no delta against the merge base — and it writes that to stderr, so an unchecked call looks silent and successful. Failing to notice means you burned a real-AWS deploy + destroy and recorded nothing: the merge is still blocked, and the natural reaction is to run the integ AGAIN rather than to git fetch origin. Run from the PR's own worktree on the PR branch, and if it exits 2, fix the base ref rather than re-running the integ.

   If any of the above failed, do NOT set the marker — that is the whole point of the gate. The hook .claude/hooks/integ-destroy-gate.sh will block gh pr merge for any PR that touches deletion-related code (see .markgate.yml integ-destroy.include) until this marker is fresh, so a destroy-untested change physically cannot reach main.

10. Set the integ-local markgate marker (only for local-* tests, on full clean success):

    When the integ test name starts with local- (i.e. local-invoke, local-start-api, local-run-task, local-invoke-container, local-invoke-from-state, local-invoke-layers, local-invoke-python / -ruby / -java / -dotnet / -provided, local-start-api-cors, or any future local-* test), ALSO set the integ-local marker after a clean Docker run.

    Required cleanup verification BEFORE setting the marker (in addition to the conditions above for integ-destroy):

    # All three queries MUST return empty. If any lists IDs, the
    # marker is NOT set and the user is shown the orphan container /
    # network IDs to clean up via `docker rm -f` / `docker network rm`.
    # `-a`, not a bare `docker ps`: a print-and-exit task container is already
    # `Exited` by the time this runs, so a running-only sweep reports clean over
    # a real orphan. That blind spot let one survive every `local-run-task-from-state`
    # run until it was caught twice in a row while gating #2183.
    docker ps -a --filter name=cdkd-local- --format '{{.ID}}'
    docker network ls --filter name=cdkd-local-task- --format '{{.ID}}'
    docker network ls --filter name=cdkd-local-svc- --format '{{.ID}}'
    

    Subnet-overlap gotcha (seen 2026-07-27): cdkd local start-service creates its shared network on the FIXED subnet 169.254.171.0/24, so a local-start-* test can fail with Pool overlaps with other one on this address space even when all three queries above are empty — a foreign leftover network (e.g. cdk-local's cdkl-svc-* from a crashed run) may own that subnet. Diagnose with docker network inspect $(docker network ls -q) --format '{{.Name}} {{range .IPAM.Config}}{{.Subnet}}{{end}} {{len .Containers}}' and remove the holder ONLY when it has 0 attached containers (a non-empty one may belong to a live parallel run).

11. Set the integ-broad markgate marker (only for BROAD integ tests, on full clean success):

    The broad-integ set covers tests that exercise multi-resource deploy/destroy paths (VPC + NAT + Lambda hyperplane ENI, Custom Resource, DAG with 5+ types across 2+ levels). A test is "broad" iff its name is one of:

    bench-cdk-sample
    lambda
    microservices
    drift-revert
    drift-revert-vpc
    multi-stack-deps
    multi-resource
    remove-protection
    export
    

    Only FIVE of those nine carry a verify.sh, and from an agent session the other four cannot be run at all. Step 5's dispatch note explains why (the standard flow needs a direct cdkd deploy, which the harness's auto-approval classifier refuses), but it leaves the selection to be discovered one dead-end at a time. Measured 2026-08-26 while gating a destroy.ts change: multi-stack-deps is the obvious pick for an --all loop change and is one of the four without one.

    Runnable from a session: lambda, drift-revert, drift-revert-vpc, remove-protection, export. Human-driven shell only: bench-cdk-sample, microservices, multi-stack-deps, multi-resource.

    lambda is the cheap default — a ~100 s, 9-resource DAG across SQS / IAM / Lambda / LayerVersion / DynamoDB Table + GlobalTable, which is what makes it broad. Re-derive the split with ls tests/integration/<name>/verify.sh rather than trusting this list if a fixture has since gained one.

    (Keep this list in sync with .claude/hooks/integ-broad-gate.sh's error message and the matching memory rule feedback_cross_cutting_needs_broad_integ.md. The sync is now enforced rather than requested: tests/unit/scripts/cross-cutting-list-sync.test.ts compares all seven copies of this list, after the hook's own header comment was found sitting at 8 entries while every other copy had 9.)

    When the integ test name is in the broad set AND the destroy step finished cleanly with 0 errors / 0 orphans (= the same conditions that flip integ-destroy), ALSO record the broad-integ sentinel and flip the marker:

    # Sentinel content is informational (human-readable, helps the
    # next /verify-pr run know which integ was last run). The
    
    
        \
      > .markgate-broad-integ-test
    mise  -- markgate  integ-broad
    

12. Set the integ-schema-migration markgate marker (only for schema-v*-to-v*-migration tests, on full clean success):

    cdkd's S3 state schema is the actual user contract. A schema version bump (e.g. v5 -> v6) MUST be transparently auto-migrated by the new binary AND verified by a real-AWS integ test that proves the round-trip: deploy under vN -> swap binary -> read works against the vN state without re-deploying -> next write upgrades to vN+1 silently -> destroy clean / 0 orphans.

    A test is "schema-migration" iff its name matches the pattern schema-v<N>-to-v<N+1>-migration (e.g. schema-v5-to-v6-migration). Test fixtures live under tests/integration/schema-v<N>-to-v<N+1>-migration/.

    When the test name matches AND the destroy step finished cleanly with 0 errors / 0 orphans (= the same conditions that flip integ-destroy), ALSO record the schema-migration sentinel and flip the marker. Unlike integ-broad, this gate's marker uses src/types/state.ts as its include scope directly, so the sentinel is informational only (no separate sentinel file is needed for markgate's digest):

    mise exec -- markgate set integ-schema-migration
    

    The hook .claude/hooks/integ-schema-migration-gate.sh blocks gh pr merge for any PR that bumps the StackState.version literal type in src/types/state.ts (detected via precise gh pr diff grep — non-bump edits to state.ts pass through with no false positive) until this marker is fresh. Same 14d TTL and same "do NOT call markgate set directly to bypass" rule as the other AWS-coupled gates.

    Non-schema-migration tests do NOT set this marker — e.g. lambda flips integ-destroy + integ-broad but leaves integ-schema-migration alone. A schema-bump PR must run a test named exactly schema-v<N>-to-v<N+1>-migration to clear this gate. See memory rule feedback_schema_version_migration_integ_required.md for the full migration-test checklist + the absolute requirement that auto-migration must be transparent (no user action required on upgrade).

13. Record the run in the integ ledger (MANDATORY — every run, pass OR fail):

    docs/_generated/integ-last-run.tsv is a COMMITTED (NOT gitignored), update-type ledger — one row per test — so anyone can see when each integ last ran and whether it passed. This answers "has this been run recently?" / "this hasn't run in months, it's risky to trust" without trawling CI history, and feeds /pick-integ. Write it on EVERY /run-integ invocation, pass or fail, right after the marker steps above (or right after a failure is recorded — never skip it on failure).

    Columns (TAB-separated): test last_run_iso result duration_s flow note

    • result: PASS only when the run finished cleanly (destroy 0 errors AND 0 orphans; verify.sh exited 0) — the SAME bar as the markgate markers. Otherwise FAIL.
    • last_run_iso: date -u +%Y-%m-%dT%H:%M:%SZ (UTC). flow: verify.sh or standard.
    • duration_s: optional wall-clock seconds. note: short reason / finding one-liner.

    Append the new row, then normalize the whole file (do NOT hand-drop the old row — the normalizer collapses to the newest row per test and re-sorts): Use an ABSOLUTE path into the feature worktree for LEDGER. The session's persistent Bash cwd can silently reset to the MAIN worktree (observed right after a background integ task completes — the cwd-race signature in main-tree-git-cwd-detector.sh), and a relative docs/_generated/... write then dirties the main tree on main, which the main-tree-dirty-detector hook flags and you must then repair. Verify with pwd immediately before the write, or hardcode LEDGER=/abs/path/to/.claude/worktrees/<branch>/docs/_generated/integ-last-run.tsv.

    # ABSOLUTE path into the feature worktree (see the cwd-race note above);
    # substitute your worktree path, do not rely on the session cwd.
    LEDGER="/path/to/repo/.claude/worktrees/<branch>/docs/_generated/integ-last-run.tsv"
    # Bootstrap the header if the file is somehow absent — `>>` alone would create it
    # headerless, and the normalizer preserves whatever header it finds (none), silently
    
    [ -f  ] ||   \
       \
       \
       \
       \
       \
       \
       > 
    TEST=; TS=
    RESULT=; DUR=; FLOW=; NOTE=
            >> 
    vp run integ-ledger-normalize
    

Important

• Run /review-pr (and apply its fixes) BEFORE this skill when both are planned for the same PR. The integ-destroy / integ-broad / integ-local markers are digest-bound to their src scopes, so any post-integ review fix that touches a provider or deploy/destroy file stales the marker and forces a full re-run against real AWS. Review polish first, integ last (memory rule feedback_apply_review_polish_before_integ — recurred 2026-07-31 on the #1282 PR, costing a second CloudFront deploy + destroy cycle).
• Always use --region us-east-1 for integration tests
• Always destroy after deploy to avoid leftover resources
• If deploy fails, still attempt destroy to clean up partial state
• A run blocked BEFORE its assertions is not a test failure — say which it was. cdkd gc refuses while ANY stack in the state bucket holds a lock (an account-wide guard, deliberate and documented at its site), so on an account with parallel agent sessions a gc fixture can be stopped before it reaches a single gc assertion. That happened twice in a row on 2026-08-19, from two different foreign stacks in a different region than the one under test. Both are recorded as FAIL rows because the bar is exit-code-based, and both notes say plainly that the fix was not at fault. When a run dies before its assertions, do all three: clean up as usual (an aborted run still leaks — one of these left a state file, a Lambda and an IAM role behind), write the ledger note so it names the blocker rather than implying the change is broken, and WAIT for the blocker to clear rather than forcing past it. Never cdkd force-unlock a lock you did not take: it belongs to another session's in-flight deploy.
• Never report success based on a successful deploy alone — destroy must complete and orphan check must pass
• Never bypass this skill by calling cdkd deploy / cdkd destroy directly from a shell — the orphan-cleanup contract above is part of the integration test, not optional