Run integration tests against a real AWS account. These tests deploy actual AWS resources, verify them, and clean up.
-
Build first: Run vp run build to ensure dist/ is up to date.
-
List available tests: Run ls tests/integration/ to discover all test directories dynamically. Do NOT rely on a hardcoded list.
-
Determine state bucket: Resolve dynamically via aws sts get-caller-identity --query Account --output text to get the account ID, then construct cdkd-state-{accountId} (region-free, the current default since PR #62 / v0.11.0). If that bucket doesn't exist, fall back to the legacy cdkd-state-{accountId}-us-east-1 and note the deprecation in the report.
-
Pre-flight orphan scan (mandatory — fail fast on prior-run leftovers instead of going through CREATE + rollback):
Before invoking deploy, scan AWS for resources matching the stack name from this test that have no business existing yet. The scenario this catches: a previous integration run was killed mid-deploy, leaving orphan Event Source Mappings / Lambda functions / ENIs / IAM roles whose names match the stack about to be deployed. cdkd's diff calculation does NOT see these (they're not in state), so the deploy attempts CREATE — which collides with the orphans, fails immediately with ResourceAlreadyExists, and forces a CREATE-then-rollback cycle. Failing at the start is much cheaper than partway through.
Synth first (without deploy) to learn the stack name and the resource types in the template, then for each scenario in scope run a targeted scan:
aws s3 ls s3://<bucket>/cdkd/<StackName>/ --region us-east-1
aws iam list-roles --query 'Roles[?contains(RoleName, `<StackName>`)].RoleName' --output text
aws lambda list-functions --region us-east-1 \
--query 'Functions[?contains(FunctionName, `<StackName>`)].FunctionName' --output text
aws lambda list-event-source-mappings --region us-east-1 \
--query 'EventSourceMappings[?contains(FunctionArn, `<StackName>`)].[UUID,FunctionArn]' --output text
aws ec2 describe-network-interfaces --region us-east-1 \
--filters "Name=description,Values=AWS Lambda VPC ENI-<StackName>*" \
--query 'NetworkInterfaces[].[NetworkInterfaceId,Status]' --output text
If anything is found, abort the test run with a clear report listing the orphans and the cleanup commands the user should run (aws lambda delete-event-source-mapping --uuid …, cdkd state destroy <StackName> --yes, etc.) — do NOT proceed with deploy. Resuming on top of orphans is the failure mode this step exists to prevent.
If nothing is found, the deploy can proceed cleanly.
-
Run the test(s):
Dispatch: a verify.sh in tests/integration/<test-name>/ is for tests with non-standard flows (drift-injection, multi-stage validation, etc.) — the script owns its own deploy + verify + destroy cycle. The standard flow below is for plain "deploy this stack and destroy it" smoke tests. Pre-flight (step 4) and the post-run verification (steps 6 + 7) apply to BOTH paths — they are the safety net that catches a buggy verify.sh leaking resources.
- Navigate to
tests/integration/<test-name>/
- Ensure dependencies:
npm install if node_modules doesn't exist
- If
tests/integration/<test-name>/verify.sh exists, run it instead of the standard flow:
AWS_REGION=us-east-1 STATE_BUCKET=<bucket> bash verify.sh- The script is responsible for its own deploy + destroy cycle. Steps 6 (verify cleanup) and 7 (auto-cleanup orphans) STILL run after — do not skip them.
- Propagate the script's exit code: a non-zero exit must drive the skill into the failure path so step 7's auto-cleanup fires. Do NOT swallow
verify.sh failures.
- Skip the synth / deploy / destroy commands below (the script does its own).
- Otherwise (no
verify.sh), run the standard flow (synth → deploy → destroy):
- Run synth:
node ../../../dist/cli.js synth --region us-east-1
- Detect multi-stack apps: read the synth output. If it lists more
than one stack (e.g.
multi-stack-deps, composite-stack,
cross-stack-references), pass --all to deploy and destroy.
Without --all, deploy/destroy will fail with Multiple stacks found: ... Specify stack name(s) or use --all.
- Run deploy:
node ../../../dist/cli.js deploy [--all] [<extra-deploy-args>] --region us-east-1 --state-bucket <bucket> --verbose
- When
--deploy-args "<args>" was passed to the skill, splice those args into the deploy invocation verbatim. Don't apply them to destroy.
- Run destroy:
node ../../../dist/cli.js destroy [--all] --region us-east-1 --state-bucket <bucket> --force
-
Verify cleanup:
- Check
aws s3 ls s3://<bucket>/cdkd/ --region us-east-1 to confirm no leftover state
- Also verify actual AWS resources are gone by checking with stack name prefix filters. Get stack names from the synth output, then for each stack name query AWS APIs filtered by that prefix:
aws iam list-roles --query 'Roles[?contains(RoleName, \{StackName}`)].RoleName'`aws lambda list-functions --region us-east-1 --query 'Functions[?contains(FunctionName, \{StackName}`)].FunctionName'`aws s3api list-buckets --query 'Buckets[?contains(Name, \{stackName-lowercase}`)].Name'`aws ecr describe-repositories --region us-east-1 --query 'repositories[?contains(repositoryName, \{stackName-lowercase}`)].repositoryName'`aws dynamodb list-tables --region us-east-1 --query 'TableNames[?contains(@, \{StackName}`)]'`- For VPC-based tests also check:
aws ec2 describe-vpcs --filters "Name=tag:Name,Values={StackName}/Vpc" ...
- Only check resource types relevant to the test being run
-
Auto-cleanup orphans (mandatory when destroy didn't fully succeed):
Trigger this step whenever any of the following is true:
- The
destroy step in step 5 reported a non-zero error count (e.g. "X failed to delete")
- Step 6 found a leftover S3 state file
- Step 6 found any AWS resource matching the stack name prefix
What to do:
- For VPC-attached Lambda failures (the most common pattern), the typical orphan set is, in delete order:
- Lambda hyperplane ENIs (
aws ec2 describe-network-interfaces --filters "Name=vpc-id,Values=<vpc>" → aws ec2 delete-network-interface). Some may be in-use initially — re-poll until they go available, then delete.
- SecurityGroups (
aws ec2 delete-security-group) — must come after the ENIs that reference them are gone.
- Subnets (
aws ec2 delete-subnet) — must come after every ENI in them is gone.
- VPC (
aws ec2 delete-vpc) — last.
- For S3 state orphans:
aws s3 rm s3://<bucket>/cdkd/<StackName>/ --recursive. (cdkd state orphan <StackName> is the cdkd-native equivalent and also handles the lock key.)
- For other resource types, infer the right delete order from CloudFormation dependency rules (children before parents).
- Always specify the correct region (
--region).
- Re-run step 6 after cleanup to confirm zero orphans remain.
Never end the run with orphan resources still present in AWS. Cost (NAT GW alone is ~$1/hr) and account hygiene make this non-negotiable. If a resource genuinely cannot be deleted after reasonable retries, surface it to the user with the exact ID, region, and what was tried — but only after the auto-cleanup pass.
-
Report results: Show pass/fail for each test, including resource counts and timing. Always state explicitly "destroy completed: 0 errors, 0 orphans" or itemize what remained / what was force-cleaned.
-
Set the integ-destroy markgate marker (only on full clean success):
When — and ONLY when — all of the following hold:
- the destroy step finished with 0 errors,
- step 6 found 0 leftover resources,
- step 7 was either skipped (because nothing to clean up) or completed with the post-cleanup re-check showing 0 orphans,
record the gate so subsequent gh pr merge calls are unblocked:
mise exec -- markgate set integ-destroy
If any of the above failed, do NOT set the marker — that is the
whole point of the gate. The hook
.claude/hooks/integ-destroy-gate.sh will block gh pr merge for
any PR that touches deletion-related code (see .markgate.yml
integ-destroy.include) until this marker is fresh, so a
destroy-untested change physically cannot reach main.
-
Set the integ-local markgate marker (only for local-* tests, on full clean success):
When the integ test name starts with local- (i.e. local-invoke,
local-start-api, local-run-task, local-invoke-container,
local-invoke-from-state, local-invoke-layers,
local-invoke-python / -ruby / -java / -dotnet / -provided,
local-start-api-cors, or any future local-* test), ALSO set
the integ-local marker after a clean Docker run.
Required cleanup verification BEFORE setting the marker (in
addition to the conditions above for integ-destroy):
docker ps --filter name=cdkd-local- --format '{{.ID}}'
docker network ls --filter name=cdkd-local-task- --format '{{.ID}}'
When BOTH return empty AND the integ test exited cleanly:
mise exec -- markgate set integ-local
The hook .claude/hooks/integ-local-gate.sh blocks gh pr merge
(and git merge) for any PR that touches cdkd local * code
(see .markgate.yml integ-local.include) until this marker is
fresh, so an unverified local-execution change physically cannot
reach main. Same TTL (14d) and same "do NOT call markgate set
directly to bypass" rule as integ-destroy.
Note: Non-local-* tests (e.g. bench-cdk-sample, lambda)
leave the integ-local marker untouched. The two markers are
independent — a lambda integ run does not refresh integ-local,
and a local-invoke run does not refresh integ-destroy unless
its verify.sh also exercises a real-AWS deploy + destroy (the
local-invoke-from-state test does, so it can set BOTH).
-
Set the integ-broad markgate marker (only for BROAD integ tests, on full clean success):
The broad-integ set covers tests that exercise multi-resource
deploy/destroy paths (VPC + NAT + Lambda hyperplane ENI, Custom
Resource, DAG with 5+ types across 2+ levels). A test is "broad"
iff its name is one of:
bench-cdk-sample
lambda
microservices
drift-revert
drift-revert-vpc
multi-stack-deps
multi-resource
remove-protection
export
(Keep this list in sync with .claude/hooks/integ-broad-gate.sh's
error message and the matching memory rule
feedback_cross_cutting_needs_broad_integ.md.)
When the integ test name is in the broad set AND the destroy step
finished cleanly with 0 errors / 0 orphans (= the same conditions
that flip integ-destroy), ALSO record the broad-integ sentinel
and flip the marker:
printf '%s ran at %s\n' "<test-name>" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
> .markgate-broad-integ-test
mise exec -- markgate set integ-broad
The hook .claude/hooks/integ-broad-gate.sh blocks gh pr merge
for any PR that touches cross-cutting deploy/destroy code (see the
hook's CROSS_CUTTING_REGEX for the canonical list) until this
marker is fresh. Same 14d TTL and same "do NOT call markgate set
directly to bypass" rule as the other AWS-coupled gates.
Narrow feature integs do NOT set this marker — e.g.
import-value-strong-ref flips integ-destroy but leaves
integ-broad alone. This is the structural fix for the PR #348
incident: a 2-stack S3+SSM feature fixture is sufficient for the
feature's correctness, but a cross-cutting change to
src/deployment/deploy-engine.ts etc. needs the broader VPC /
Lambda / multi-resource coverage that only the broad set provides.
-
Set the integ-schema-migration markgate marker (only for
schema-v*-to-v*-migration tests, on full clean success):
cdkd's S3 state schema is the actual user contract. A schema
version bump (e.g. v5 -> v6) MUST be transparently auto-migrated
by the new binary AND verified by a real-AWS integ test that
proves the round-trip: deploy under vN -> swap binary -> read
works against the vN state without re-deploying -> next write
upgrades to vN+1 silently -> destroy clean / 0 orphans.
A test is "schema-migration" iff its name matches the pattern
schema-v<N>-to-v<N+1>-migration (e.g.
schema-v5-to-v6-migration). Test fixtures live under
tests/integration/schema-v<N>-to-v<N+1>-migration/.
When the test name matches AND the destroy step finished cleanly
with 0 errors / 0 orphans (= the same conditions that flip
integ-destroy), ALSO record the schema-migration sentinel and
flip the marker. Unlike integ-broad, this gate's marker uses
src/types/state.ts as its include scope directly, so the
sentinel is informational only (no separate sentinel file is
needed for markgate's digest):
mise exec -- markgate set integ-schema-migration
The hook .claude/hooks/integ-schema-migration-gate.sh blocks
gh pr merge for any PR that bumps the StackState.version
literal type in src/types/state.ts (detected via precise gh pr diff grep — non-bump edits to state.ts pass through with no
false positive) until this marker is fresh. Same 14d TTL and
same "do NOT call markgate set directly to bypass" rule as the
other AWS-coupled gates.
Non-schema-migration tests do NOT set this marker — e.g.
lambda flips integ-destroy + integ-broad but leaves
integ-schema-migration alone. A schema-bump PR must run a
test named exactly schema-v<N>-to-v<N+1>-migration to clear
this gate. See memory rule
feedback_schema_version_migration_integ_required.md for the
full migration-test checklist + the absolute requirement that
auto-migration must be transparent (no user action required
on upgrade).
-
Record the run in the integ ledger (MANDATORY — every run, pass OR fail):
docs/_generated/integ-last-run.tsv is a COMMITTED (NOT gitignored), update-type
ledger — one row per test — so anyone can see when each integ last ran and whether
it passed. This answers "has this been run recently?" / "this hasn't run in months,
it's risky to trust" without trawling CI history, and feeds /pick-integ. Write it
on EVERY /run-integ invocation, pass or fail, right after the marker steps above
(or right after a failure is recorded — never skip it on failure).
Columns (TAB-separated): test last_run_iso result duration_s flow note
result: PASS only when the run finished cleanly (destroy 0 errors AND 0 orphans;
verify.sh exited 0) — the SAME bar as the markgate markers. Otherwise FAIL.last_run_iso: date -u +%Y-%m-%dT%H:%M:%SZ (UTC). flow: verify.sh or standard.duration_s: optional wall-clock seconds. note: short reason / finding one-liner.
Portable update (do NOT use grep -P — unavailable on macOS BSD grep; use awk to drop
the test's old row, then append the new one):
LEDGER="docs/_generated/integ-last-run.tsv"
[ -f "$LEDGER" ] || printf '# integ-last-run ledger (update-type: one row per test). cols: test\tlast_run_iso\tresult\tduration_s\tflow\tnote\n' > "$LEDGER"
TEST="<test-name>"; TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
RESULT="PASS"; DUR="<seconds>"; FLOW="verify.sh"; NOTE="rc ok, orph clean"
tmp="$(mktemp)"; awk -F'\t' -v t="$TEST" '$1!=t' "$LEDGER" > "$tmp" && mv "$tmp" "$LEDGER"
printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$TEST" "$TS" "$RESULT" "$DUR" "$FLOW" "$NOTE" >> "$LEDGER"
Commit the ledger update with the branch's changes (it is part of the integ run record;
the file is intentionally committed so the last-run history is shared across sessions).
