run-integ

name	run-integ
description	Run integration tests (deploy + destroy) against real AWS. Use when you need to verify cdkd works end-to-end with actual AWS resources.
argument-hint	<test-name|all> [--synth-only] [--no-destroy]

Integration Test Runner

Run integration tests against a real AWS account. These tests deploy actual AWS resources, verify them, and clean up.

Arguments

• test-name: Which test to run. Run ls tests/integration/ to see all available tests. If not specified, use the AskUserQuestion tool to ask which test to run, showing the available options.
• all: Run all tests
• --synth-only: Only run synthesis, skip deploy/destroy
• --no-destroy: Deploy but don't destroy (for debugging)
• --deploy-args "<args>": Forward extra arguments to the cdkd deploy invocation (e.g. --deploy-args "--aggressive-vpc-parallel"). Use when validating an opt-in deploy flag end-to-end against bench-cdk-sample etc. — the destroy step is unaffected (opt-in flags so far are deploy-only).

Steps

1. Build first: Run vp run build to ensure dist/ is up to date.

2. List available tests: Run ls tests/integration/ to discover all test directories dynamically. Do NOT rely on a hardcoded list.

3. Determine state bucket: Resolve dynamically via aws sts get-caller-identity --query Account --output text to get the account ID, then construct cdkd-state-{accountId} (region-free, the current default since PR #62 / v0.11.0). If that bucket doesn't exist, fall back to the legacy cdkd-state-{accountId}-us-east-1 and note the deprecation in the report.

4. Pre-flight orphan scan (mandatory — fail fast on prior-run leftovers instead of going through CREATE + rollback):

   Before invoking deploy, scan AWS for resources matching the stack name from this test that have no business existing yet. The scenario this catches: a previous integration run was killed mid-deploy, leaving orphan Event Source Mappings / Lambda functions / ENIs / IAM roles whose names match the stack about to be deployed. cdkd's diff calculation does NOT see these (they're not in state), so the deploy attempts CREATE — which collides with the orphans, fails immediately with ResourceAlreadyExists, and forces a CREATE-then-rollback cycle. Failing at the start is much cheaper than partway through.

   Synth first (without deploy) to learn the stack name and the resource types in the template, then for each scenario in scope run a targeted scan:

   # Always run these (cheap, broadly applicable):
   aws s3 ls s3://<bucket>/cdkd/<StackName>/ --region us-east-1
   aws iam list-roles --query 'Roles[?contains(RoleName, `<StackName>`)].RoleName' --output text
   aws lambda list-functions --region us-east-1 \
     --query 'Functions[?contains(FunctionName, `<StackName>`)].FunctionName' --output text
   
   # Run when the template uses Lambda EventSourceMapping (the orphan ESM
   # case bit cdkd in 2026-05-02: AlreadyExists + rollback cycle on a fresh deploy):
   aws lambda list-event-source-mappings --region us-east-1 \
     --query 'EventSourceMappings[?contains(FunctionArn, `<StackName>`)].[UUID,FunctionArn]' --output text
   
   # Run when the template uses VPC + Lambda VpcConfig (hyperplane ENIs
   # outlive their function):
   aws ec2 describe-network-interfaces --region us-east-1 \
     --filters "Name=description,Values=AWS Lambda VPC ENI-<StackName>*" \
     --query 'NetworkInterfaces[].[NetworkInterfaceId,Status]' --output text
   

   If anything is found, abort the test run with a clear report listing the orphans and the cleanup commands the user should run (aws lambda delete-event-source-mapping --uuid …, cdkd state destroy <StackName> --yes, etc.) — do NOT proceed with deploy. Resuming on top of orphans is the failure mode this step exists to prevent.

   If nothing is found, the deploy can proceed cleanly.

5. Run the test(s):

   Dispatch: a verify.sh in tests/integration/<test-name>/ is for tests with non-standard flows (drift-injection, multi-stage validation, etc.) — the script owns its own deploy + verify + destroy cycle. The standard flow below is for plain "deploy this stack and destroy it" smoke tests. Pre-flight (step 4) and the post-run verification (steps 6 + 7) apply to BOTH paths — they are the safety net that catches a buggy verify.sh leaking resources.

   • Navigate to tests/integration/<test-name>/
   • Ensure dependencies: npm install if node_modules doesn't exist
   • If tests/integration/<test-name>/verify.sh exists, run it instead of the standard flow:
     • AWS_REGION=us-east-1 STATE_BUCKET=<bucket> bash verify.sh
     • The script is responsible for its own deploy + destroy cycle. Steps 6 (verify cleanup) and 7 (auto-cleanup orphans) STILL run after — do not skip them.
     • Propagate the script's exit code: a non-zero exit must drive the skill into the failure path so step 7's auto-cleanup fires. Do NOT swallow verify.sh failures.
     • Skip the synth / deploy / destroy commands below (the script does its own).
   • Otherwise (no verify.sh), run the standard flow (synth → deploy → destroy):
     • Run synth: node ../../../dist/cli.js synth --region us-east-1
     • Detect multi-stack apps: read the synth output. If it lists more than one stack (e.g. multi-stack-deps, composite-stack, cross-stack-references), pass --all to deploy and destroy. Without --all, deploy/destroy will fail with Multiple stacks found: ... Specify stack name(s) or use --all.
     • Run deploy: node ../../../dist/cli.js deploy [--all] [<extra-deploy-args>] --region us-east-1 --state-bucket <bucket> --verbose
       • When --deploy-args "<args>" was passed to the skill, splice those args into the deploy invocation verbatim. Don't apply them to destroy.
     • Run destroy: node ../../../dist/cli.js destroy [--all] --region us-east-1 --state-bucket <bucket> --force

6. Verify cleanup:

   • Check aws s3 ls s3://<bucket>/cdkd/ --region us-east-1 to confirm no leftover state
   • Also verify actual AWS resources are gone by checking with stack name prefix filters. Get stack names from the synth output, then for each stack name query AWS APIs filtered by that prefix:
     • aws iam list-roles --query 'Roles[?contains(RoleName, \{StackName}`)].RoleName'`
     • aws lambda list-functions --region us-east-1 --query 'Functions[?contains(FunctionName, \{StackName}`)].FunctionName'`
     • aws s3api list-buckets --query 'Buckets[?contains(Name, \{stackName-lowercase}`)].Name'`
     • aws ecr describe-repositories --region us-east-1 --query 'repositories[?contains(repositoryName, \{stackName-lowercase}`)].repositoryName'`
     • aws dynamodb list-tables --region us-east-1 --query 'TableNames[?contains(@, \{StackName}`)]'`
     • For VPC-based tests also check: aws ec2 describe-vpcs --filters "Name=tag:Name,Values={StackName}/Vpc" ...
   • Only check resource types relevant to the test being run

7. Auto-cleanup orphans (mandatory when destroy didn't fully succeed):

   Trigger this step whenever any of the following is true:

   • The destroy step in step 5 reported a non-zero error count (e.g. "X failed to delete")
   • Step 6 found a leftover S3 state file
   • Step 6 found any AWS resource matching the stack name prefix

   What to do:

   • For VPC-attached Lambda failures (the most common pattern), the typical orphan set is, in delete order:
     1. Lambda hyperplane ENIs (aws ec2 describe-network-interfaces --filters "Name=vpc-id,Values=<vpc>" → aws ec2 delete-network-interface). Some may be in-use initially — re-poll until they go available, then delete.
     2. SecurityGroups (aws ec2 delete-security-group) — must come after the ENIs that reference them are gone.
     3. Subnets (aws ec2 delete-subnet) — must come after every ENI in them is gone.
     4. VPC (aws ec2 delete-vpc) — last.
   • For S3 state orphans: aws s3 rm s3://<bucket>/cdkd/<StackName>/ --recursive. (cdkd state orphan <StackName> is the cdkd-native equivalent and also handles the lock key.)
   • For other resource types, infer the right delete order from CloudFormation dependency rules (children before parents).
   • Always specify the correct region (--region).
   • Re-run step 6 after cleanup to confirm zero orphans remain.

   Never end the run with orphan resources still present in AWS. Cost (NAT GW alone is ~$1/hr) and account hygiene make this non-negotiable. If a resource genuinely cannot be deleted after reasonable retries, surface it to the user with the exact ID, region, and what was tried — but only after the auto-cleanup pass.

8. Report results: Show pass/fail for each test, including resource counts and timing. Always state explicitly "destroy completed: 0 errors, 0 orphans" or itemize what remained / what was force-cleaned.

9. Set the integ-destroy markgate marker (only on full clean success):

   When — and ONLY when — all of the following hold:

   • the destroy step finished with 0 errors,
   • step 6 found 0 leftover resources,
   • step 7 was either skipped (because nothing to clean up) or completed with the post-cleanup re-check showing 0 orphans,

   record the gate so subsequent gh pr merge calls are unblocked:

   mise exec -- markgate set integ-destroy
   

   If any of the above failed, do NOT set the marker — that is the whole point of the gate. The hook .claude/hooks/integ-destroy-gate.sh will block gh pr merge for any PR that touches deletion-related code (see .markgate.yml integ-destroy.include) until this marker is fresh, so a destroy-untested change physically cannot reach main.

10. Set the integ-local markgate marker (only for local-* tests, on full clean success):

    When the integ test name starts with local- (i.e. local-invoke, local-start-api, local-run-task, local-invoke-container, local-invoke-from-state, local-invoke-layers, local-invoke-python / -ruby / -java / -dotnet / -provided, local-start-api-cors, or any future local-* test), ALSO set the integ-local marker after a clean Docker run.

    Required cleanup verification BEFORE setting the marker (in addition to the conditions above for integ-destroy):

    # Both queries MUST return empty. If either lists any IDs, the
    # marker is NOT set and the user is shown the orphan container /
    # network IDs to clean up via `docker rm -f` / `docker network rm`.
    docker ps --filter name=cdkd-local- --format '{{.ID}}'
    docker network ls --filter name=cdkd-local-task- --format '{{.ID}}'
    

    When BOTH return empty AND the integ test exited cleanly:

    mise exec -- markgate set integ-local
    

    The hook .claude/hooks/integ-local-gate.sh blocks gh pr merge (and git merge) for any PR that touches cdkd local * code (see .markgate.yml integ-local.include) until this marker is fresh, so an unverified local-execution change physically cannot reach main. Same TTL (14d) and same "do NOT call markgate set directly to bypass" rule as integ-destroy.

    Note: Non-local-* tests (e.g. bench-cdk-sample, lambda) leave the integ-local marker untouched. The two markers are independent — a lambda integ run does not refresh integ-local, and a local-invoke run does not refresh integ-destroy unless its verify.sh also exercises a real-AWS deploy + destroy (the local-invoke-from-state test does, so it can set BOTH).

11. Set the integ-broad markgate marker (only for BROAD integ tests, on full clean success):

    The broad-integ set covers tests that exercise multi-resource deploy/destroy paths (VPC + NAT + Lambda hyperplane ENI, Custom Resource, DAG with 5+ types across 2+ levels). A test is "broad" iff its name is one of:

    bench-cdk-sample
    lambda
    microservices
    drift-revert
    drift-revert-vpc
    multi-stack-deps
    multi-resource
    remove-protection
    export
    

    (Keep this list in sync with .claude/hooks/integ-broad-gate.sh's error message and the matching memory rule feedback_cross_cutting_needs_broad_integ.md.)

    When the integ test name is in the broad set AND the destroy step finished cleanly with 0 errors / 0 orphans (= the same conditions that flip integ-destroy), ALSO record the broad-integ sentinel and flip the marker:

    # Sentinel content is informational (human-readable, helps the
    # next /verify-pr run know which integ was last run). The
    # `integ-broad` markgate gate's include scope is just this file,
    # so writing the test name flips its digest naturally.
    printf '%s ran at %s\n' "<test-name>" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
      > .markgate-broad-integ-test
    mise exec -- markgate set integ-broad
    

    The hook .claude/hooks/integ-broad-gate.sh blocks gh pr merge for any PR that touches cross-cutting deploy/destroy code (see the hook's CROSS_CUTTING_REGEX for the canonical list) until this marker is fresh. Same 14d TTL and same "do NOT call markgate set directly to bypass" rule as the other AWS-coupled gates.

    Narrow feature integs do NOT set this marker — e.g. import-value-strong-ref flips integ-destroy but leaves integ-broad alone. This is the structural fix for the PR #348 incident: a 2-stack S3+SSM feature fixture is sufficient for the feature's correctness, but a cross-cutting change to src/deployment/deploy-engine.ts etc. needs the broader VPC / Lambda / multi-resource coverage that only the broad set provides.

12. Set the integ-schema-migration markgate marker (only for schema-v*-to-v*-migration tests, on full clean success):

    cdkd's S3 state schema is the actual user contract. A schema version bump (e.g. v5 -> v6) MUST be transparently auto-migrated by the new binary AND verified by a real-AWS integ test that proves the round-trip: deploy under vN -> swap binary -> read works against the vN state without re-deploying -> next write upgrades to vN+1 silently -> destroy clean / 0 orphans.

    A test is "schema-migration" iff its name matches the pattern schema-v<N>-to-v<N+1>-migration (e.g. schema-v5-to-v6-migration). Test fixtures live under tests/integration/schema-v<N>-to-v<N+1>-migration/.

    When the test name matches AND the destroy step finished cleanly with 0 errors / 0 orphans (= the same conditions that flip integ-destroy), ALSO record the schema-migration sentinel and flip the marker. Unlike integ-broad, this gate's marker uses src/types/state.ts as its include scope directly, so the sentinel is informational only (no separate sentinel file is needed for markgate's digest):

    mise exec -- markgate set integ-schema-migration
    

    The hook .claude/hooks/integ-schema-migration-gate.sh blocks gh pr merge for any PR that bumps the StackState.version literal type in src/types/state.ts (detected via precise gh pr diff grep — non-bump edits to state.ts pass through with no false positive) until this marker is fresh. Same 14d TTL and same "do NOT call markgate set directly to bypass" rule as the other AWS-coupled gates.

    Non-schema-migration tests do NOT set this marker — e.g. lambda flips integ-destroy + integ-broad but leaves integ-schema-migration alone. A schema-bump PR must run a test named exactly schema-v<N>-to-v<N+1>-migration to clear this gate. See memory rule feedback_schema_version_migration_integ_required.md for the full migration-test checklist + the absolute requirement that auto-migration must be transparent (no user action required on upgrade).

13. Record the run in the integ ledger (MANDATORY — every run, pass OR fail):

    docs/_generated/integ-last-run.tsv is a COMMITTED (NOT gitignored), update-type ledger — one row per test — so anyone can see when each integ last ran and whether it passed. This answers "has this been run recently?" / "this hasn't run in months, it's risky to trust" without trawling CI history, and feeds /pick-integ. Write it on EVERY /run-integ invocation, pass or fail, right after the marker steps above (or right after a failure is recorded — never skip it on failure).

    Columns (TAB-separated): test last_run_iso result duration_s flow note

    • result: PASS only when the run finished cleanly (destroy 0 errors AND 0 orphans; verify.sh exited 0) — the SAME bar as the markgate markers. Otherwise FAIL.
    • last_run_iso: date -u +%Y-%m-%dT%H:%M:%SZ (UTC). flow: verify.sh or standard.
    • duration_s: optional wall-clock seconds. note: short reason / finding one-liner.

    Portable update (do NOT use grep -P — unavailable on macOS BSD grep; use awk to drop the test's old row, then append the new one):

    LEDGER="docs/_generated/integ-last-run.tsv"
    [ -f "$LEDGER" ] || printf '# integ-last-run ledger (update-type: one row per test). cols: test\tlast_run_iso\tresult\tduration_s\tflow\tnote\n' > "$LEDGER"
    TEST="<test-name>"; TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
    RESULT="PASS"; DUR="<seconds>"; FLOW="verify.sh"; NOTE="rc ok, orph clean"
    tmp="$(mktemp)"; awk -F'\t' -v t="$TEST" '$1!=t' "$LEDGER" > "$tmp" && mv "$tmp" "$LEDGER"
    printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$TEST" "$TS" "$RESULT" "$DUR" "$FLOW" "$NOTE" >> "$LEDGER"
    

    Commit the ledger update with the branch's changes (it is part of the integ run record; the file is intentionally committed so the last-run history is shared across sessions).

Important

• Always use --region us-east-1 for integration tests
• Always destroy after deploy to avoid leftover resources
• If deploy fails, still attempt destroy to clean up partial state
• Never report success based on a successful deploy alone — destroy must complete and orphan check must pass
• Never bypass this skill by calling cdkd deploy / cdkd destroy directly from a shell — the orphan-cleanup contract above is part of the integration test, not optional