run-integ

name	run-integ
description	Run an integration test (Docker-based, optionally AWS-deploy-backed for the fixtures that own real AWS resources) and refresh the `integ` markgate marker on a clean run.
argument-hint	<test-name>

Integration Test Runner

Run an integration test against real Docker (and, for the fixtures that own real AWS resources, a real CloudFormation stack deployed via the upstream cdk CLI). Which fixtures those are is DERIVED by tests/integration/_lib/aws-orphan-sweep.sh, never globbed from the name.

cdk-local is a local-execution CLI — it does NOT deploy resources itself. The only AWS-side activity in any integ test is when a fixture's verify.sh invokes the upstream cdk deploy (or, for local-invoke-agentcore-froms3, aws s3api create-bucket) to create a target stack for --from-cfn-stack to point at. Cleanup is always done by the fixture's own verify.sh.

Arguments

• test-name: Which test to run. Run ls tests/integration/ to see all available tests. If not specified, use the AskUserQuestion tool to ask which test to run, showing the available options.

Steps

1. Build first: Run vp run build to ensure dist/cli.js is up to date. The fixture's verify.sh resolves the binary via node ../../../dist/cli.js, so source changes without a build have no runtime effect.

2. Resolve the fixture path: tests/integration/<test-name>/. Confirm verify.sh exists; if not, the test does not have a Docker-driven flow yet and this skill exits with a clear error pointing the user at the missing script.

3. Pre-flight Docker sweep: docker ps --filter name=cdkl- -q | wc -l and docker network ls --filter name=cdkl-task- -q | wc -l should both return 0. If either is non-zero, abort and ask the user to run /cleanup first — running on top of orphans causes name collisions and confusing failures.

4. AWS pre-flight sweep — run it for EVERY fixture, unconditionally:

   bash tests/integration/_lib/aws-orphan-sweep.sh <test-name>; rc=$?
   

   Do NOT first decide whether the fixture is AWS-resource-owning. That decision has silently excluded a resource-owning fixture twice (*-from-cfn-stack missed three, and the widened *-from-cfn* still missed local-invoke-assume-role, which deploys a stack containing an IAM role), and the script now makes it internally, from one predicate, against the fixture's own verify.sh. A fixture that owns nothing makes no AWS call and exits 0. aws-orphan-sweep.sh --list-owners prints the derived set if you want to see it.

   Also confirm the deploy toolchain is present — the sweep checks that AWS is reachable, not that the fixture can deploy, and this was dropped when the recipe became a script:

   which cdk        # the upstream CDK CLI a *-from-cfn-stack fixture shells out to
   

   Only needed when the sweep reported the fixture as AWS-owning (rc 0 with a fixture= line rather than the "owns no real AWS resource" line).

   Gate on the exit code. Do not read the output and judge. That is the whole point of go-to-k/cdk-local#601 — the previous recipe was prose whose error branches nobody executed, and it carried four instances of one defect class in a single PR (a bare stack name that could never match, a /cdkl scope that listed every lane, a filter that degraded to contains(Name,'-') over the whole ACCOUNT, and two stack scans with no guard at all, where a wrong cwd produced a false clean on the PRIMARY resource):

   rc	meaning	what to do
   0	clean	proceed to step 5
   1	usage / internal error	STOP. Nothing was concluded. Read the FATAL line.
   2	orphan found	STOP. Remediate — see below.
   3	indeterminate	STOP. A query could not be performed (no credentials, aws missing, an unrecognized error). This is NOT clean.
   4	report-only	STOP and check by hand. An UNATTRIBUTABLE resource matched — see below.

5. Run the test: bash tests/integration/<test-name>/verify.sh. Propagate the script's exit code — a non-zero exit must drive this skill into the failure path so step 7's cleanup verification fires. Do NOT swallow verify.sh failures.

6. Verify Docker cleanup (mandatory regardless of pass/fail):

   docker ps --filter name=cdkl- -q | wc -l         # must be 0
   docker network ls --filter name=cdkl-task- -q | wc -l   # must be 0
   docker network ls --filter name=cdkl-svc- -q  | wc -l   # must be 0
   

   If any are non-zero, dispatch /cleanup (no --detect-only) and re-run the checks. Never end the run with orphan Docker resources still present.

7. Verify AWS cleanup — the SAME command as step 4, with the same exit-code gate:

   bash tests/integration/_lib/aws-orphan-sweep.sh <test-name>; rc=$?
   

   This run matters more than step 4's: it comes after a long test where a session token can expire (which is exactly the rc=3 indeterminate case, NOT a clean one), and step 9 turns its verdict into a fresh integ marker.

   Same table, same remediation, same live-peer check. Re-run the sweep after any deletion and require rc=0 before continuing.

8. Report results: Show pass/fail for the test, plus a one-line cleanup summary — docker: 0 orphans, network: 0 orphans plus AWS sweep: rc=0 (clean) quoting the sweep's OWN verdict line and exit code. Quote what the script printed rather than paraphrasing it: a paraphrase is writable without having run anything, which is how a recipe reports clean while not having looked.

9. Set the integ markgate marker (only on full clean success):

   When — and ONLY when — all of the following hold:

   • the verify.sh step finished with exit code 0,
   • step 6 reports 0 docker orphans,
   • step 7's aws-orphan-sweep.sh exited 0 (any of 1 / 2 / 3 / 4 is a stop, and rc=3 in particular means the sweep could not look — not that nothing is there),

   record the gate so subsequent gh pr merge calls are unblocked:

   mise exec -- markgate set integ || echo "MARKER NOT RECORDED (rc=$?) — read the error above"
   

   Check the exit code; do not assume the set succeeded. Under hash: files this command could not fail, so it was safe to fire and forget. Under hash: diff it CAN fail, and it reports the reason on stderr — an unchecked call looks silent and successful while nothing was recorded. The failure modes and their fixes:

   • no delta against merge-base(origin/main, HEAD) — you are on the base branch. Re-run from the PR's own worktree, on the PR branch.
   • base ref "origin/main" does not resolve — run git fetch origin, then set again. Re-running the whole integ does NOT help; the set fails identically until the ref exists.
   • hash=diff recorded an empty in-scope delta — this is a WARNING, not a failure: the marker WAS saved and the exit code is 0. It only means the branch changes nothing under src/** / tests/integration/**.

   Confirm with mise exec -- markgate status integ (expect state: match) before reporting the run as complete. The expensive failure this prevents: a full Docker fixture run finishes, the marker is silently not recorded, the merge stays blocked, and the natural reaction is to run the integ AGAIN rather than fetch.

   If any of the above failed, do NOT set the marker — that is the whole point of the gate. The integ gate (see .markgate.yml) blocks gh pr merge for any PR that touches src/** or tests/integration/** until this marker is fresh. Set the marker from the PR's own worktree, on the PR branch: the gate uses markgate's mode, whose digest is that branch's delta against .

Important

• Never bypass this skill by invoking the fixture's verify.sh directly from a shell — the cleanup verification + markgate set are part of the contract.
• Never call markgate set integ directly to skip the verification. The marker only earns its place by completing the full sequence above.
• Always confirm the test name is on the official list (ls tests/integration/) — typos lead to confusing "no verify.sh" errors.
• The 14-day TTL on the marker (see .markgate.yml) accepts that Docker base-image behavior drifts over time even when the repo doesn't; re-running an integ after two weeks is the explicit revalidation. It is also what bounds hash: diff's accepted blind spot — a caller broken by this branch changing A while main changed B produces no delta overlap, so only the TTL forces the eventual re-run.