pick-integ

name	pick-integ
description	Recommend which integration tests to run, based on the integ ledger (staleness / last result) plus the code areas touched by recent commits. Outputs a prioritized list of `/run-integ <name>` commands. Use before a release, after a batch of merges, or when unsure what integ coverage a change needs.
argument-hint	[base-ref] (default: origin/main)

Integ Test Picker

Decide which integration tests to run right now. Running all ~95 back-to-back takes hours, so this skill ORDERS them: it ranks by two signals — how stale the last run is (from the committed ledger) and whether recent code changes touch the area a test exercises — and prints a prioritized, copy-pasteable /run-integ plan. It RECOMMENDS only; the orchestrator runs the chosen tests via /run-integ (which records each run back into the ledger).

The ranking is a running ORDER, not a budget. Per the "Cost is not a tiebreaker for verification depth" rule in CLAUDE.md → Workflow Rules, real-AWS time and spend are not grounds for dropping a candidate: when several tests plausibly cover the code a change touched, run all of them rather than the cheapest or the top-ranked one, and prefer a broad-set fixture over a narrow one whenever the change is cross-cutting. Use the priorities to decide what runs FIRST (so a partial session still covers the riskiest ground), not what gets skipped. If a plan is genuinely too long for one session, hand the remainder forward per "Running a large plan across sessions" below — do not silently truncate it, and say in the wrap-up exactly which tests were not run.

Inputs

• Optional positional base-ref (default origin/main): the diff base for "what changed recently". Use the last release tag / commit when picking post-merge; use origin/main when picking for a branch.

Data sources

1. Ledger docs/_generated/integ-last-run.tsv (committed, update-type, one row per test): columns test last_run_iso result duration_s flow note. Written by /run-integ on every run.

   • Stale: last_run_iso older than the integ-gate TTL window (14 days — past that, the gate markers themselves expire, so a clean result no longer proves today's AWS behavior). Bias the threshold down (treat as stale sooner) for AWS-coupled paths.
   • Expiring soon: age 12-14d — not stale yet, but inside the cliff window. The ledger accumulates in sweep-shaped cohorts (one sweep stamps 100+ rows with a single timestamp), so ages cluster into spikes rather than spreading out. A strict >14d boolean over a clustered distribution answers either "nothing" or "everything" and cannot express the genuinely actionable state — a large cohort about to expire simultaneously. Always report this tier: "zero stale" is technically true and practically misleading when 105 tests cross the TTL tomorrow (issue #1508).
   • Failing: result == FAIL — always a candidate (re-run after a fix, or to confirm still-broken).
   • Never-run: a directory under tests/integration/ with NO ledger row — highest staleness.

2. Recent code changes: git diff <base-ref>...HEAD --name-only (and/or git log <base-ref>..HEAD). Map changed source paths to the integ tests that exercise them (table below).

Steps

1. Discover the universe + ledger state:

   LEDGER="docs/_generated/integ-last-run.tsv"
   now=$(date -u +%s)
   # The ledger merges with the union driver (.gitattributes), so a rare
   # same-test collision can leave duplicate rows — the LAST row per test is
   # authoritative (deduped below before any staleness math). The header is
   # MULTIPLE `#` comment lines — skip them all (a single NR==1 guard used to
   # let later header lines through as phantom stale rows); DEDUPED holds
   # data rows only.
   DEDUPED="$(mktemp)"
   awk -F'\t' '/^#/{next} {last[$1]=$0} END{for (t in last) print last[t]}' "$LEDGER" > "$DEDUPED"
   # never-run: fixtures with no ledger row
   comm -23 \
     <(ls -d tests/integration/*/ | sed 's#tests/integration/##;s#/##' | sort) \
     <(awk -F'\t' '{print $1}' "$DEDUPED" | sort)
   # resolve each row's age ONCE (the date subprocess is the expensive part), then
   # slice that table for the histogram / stale / expiring-soon views below.
   AGES="$(mktemp)"
   awk -F'\t' -v now="$now" '{
     cmd="date -u -j -f %Y-%m-%dT%H:%M:%SZ \""$2"\" +%s 2>/dev/null || date -u -d \""$2"\" +%s 2>/dev/null";
     cmd | getline t; close(cmd);
     printf "%s\t%d\t%s\t%s\n",$1,int((now-t)/86400),$3,$6
   }' "$DEDUPED" > "$AGES"
   
   # age histogram — makes a clustered cohort visible even when NOTHING is >14d
   awk -F   |  -n |  -c |
     awk 
   
   
   awk -F  
   
   
   awk -F  
   
   
   awk -F   |
      -n |  -f2-
   

Running a large plan across sessions

A big plan (dozens of /run-integ calls) will NOT finish in one session: a single session's context degrades after roughly 15-20 deploy/destroy cycles, and the tell-tale symptom is garbled / malformed tool calls. Pushing past that point drops orphans and misreads results. So treat a large sweep as a multi-session relay, and make the orchestrator aware of it up front:

• Run in small batches (~4-5 tests). After EACH batch: (a) verify the account is orphan-clean — aws s3 ls s3://<bucket>/cdkd/ --recursive | grep state.json returns 0, and no live NAT / RDS / OpenSearch / Redshift / ElastiCache / EC2 — and (b) commit the ledger.
• A backgrounded batch loop can have a single test's subshell die right after node deploy returns but before node destroy (intermittent), leaving a full orphan (incl. a NAT GW). The post-batch state.json scan above is what catches it — never skip it. The orphan's stack NAME often differs from the fixture dir name; read it from the deploy log / synth.
• When context gets heavy (long transcript, ANY garbled tool call, or after ~15 tests this session): STOP cleanly — do not power through. Commit the ledger, then hand off: tell the user "ran N more (list), account clean, ~M remain — open a NEW session and say 'continue the sweep' to keep going." An in-progress sweep should also leave a project memory with the remaining list + any FAIL / orphan finding so the next session resumes fast.
• The sweep is DONE only when /pick-integ shows no stale tests left. Until then each session runs a slice and passes the baton; the committed ledger is the source of truth.
• A FAIL that is fixture staleness (AWS retired an engine / instance / node tier the fixture hardcodes — e.g. Cannot find version 17.4 for postgres) is NOT a cdkd bug; record it as such and queue a fixture version-bump follow-up rather than blocking the sweep.

Important

• This skill never runs /run-integ itself — it prints the plan; the orchestrator runs the chosen tests (serially — they share one AWS account; mind VPC/EIP/NAT account limits).
• The ledger is only as good as its discipline: it is updated by /run-integ on EVERY run (pass or fail). If a test's row looks impossibly old, it may simply not have been run via the skill — treat absent/old as stale.
• "Recently green + untouched" tests are the ones to schedule LAST, not the ones to drop: a test whose area the diff did not touch is genuinely lower-risk, but "lower-risk" orders the list rather than truncating it. Surface the count either way so the omission is visible.
• Pure docs / .claude/skills / test-only diffs usually need NO integ — say so rather than padding the list.