hunt-bugs

name	hunt-bugs
description	Proactively hunt for cdkrd bugs by deploying real CDK stacks that exercise common-but-untested AWS resources, configs, and CloudFormation notations against real AWS, then catch false positives + missed detection and fix what breaks. Use for a periodic "find latent bugs" sweep, not for verifying a specific change.
argument-hint	[area hint, e.g. 'rich S3' | 'CFn intrinsics' | 'console-edit detection']

cdkrd Bug Hunt

Find latent cdkrd bugs the way real users hit them: deploy a CDK stack that uses a resource / config / CloudFormation notation cdkrd has not exercised yet, then check it against real AWS and watch for misbehavior. cdkrd's logic is heavily unit-tested, so the remaining bugs live in the gap between its model of AWS and the actual live AWS response — only a real deploy surfaces those. Reading the source finds suspected bugs; deploying finds real ones.

This is a deliberately exploratory, possibly-expensive workflow. Cost is acceptable only because every deployed stack is deleted and verified gone — see "Cleanup is non-negotiable", which is enforced by a gate, not by trust.

Default posture: assume many latent bugs remain — sweep wide, ~5 rounds

Unless the user scopes it down, run this hunt on the working assumption that there are still plenty of latent bugs left to find — every past sweep has surfaced fresh FP/FN classes, and the fold/normalize tables are known-incomplete allowlists. So by default:

• Do ~5 rounds, not one. Each round is a fresh angle on a fresh set of common-but-untested types/configs. Treat one clean round as a signal to change the angle, not to stop early — keep going until ~5 rounds are done or the user says enough.
• Vary the lens every round — don't just deploy more resource types. Rotate through the angles that expose different bug classes: first-run undeclared FP (fold gaps), declared-tier normalization FP, missed-detection FN (mutate a declared mutable prop out of band), write-only / read-gap FN, revert non-convergence (silent no-op / husk-poisoned patch), composite-identifier read skips, and the offline corpus-mining sweep. The catalogue in "Core principles" + "Gotchas" is the menu; pick a different mix each round.
• Parallelize within the 3–4 stack cap and keep stack names unique so concurrent agents/sessions never collide.

"5 rounds, many angles, latent bugs assumed" is the DEFAULT — a narrower scope happens only when the user asks for it.

Goal: filing issues vs. fixing — ASK at run time unless told

The end state of a hunt is not fixed. It can stop at filing GitHub issues for what it finds, or go all the way to fixing + PR + merge. These are very different in cost, blast radius, and collision risk with parallel agents. So unless the user has explicitly stated the goal in their invocation, ASK them at the start of the run which they want — do not assume. Typical options to offer:

• Issue-only — investigate, live-verify, harvest corpus, and FILE well-scoped issues (with the repro + recommended fold/fix), but do NOT change src/. Best when other agents are working filed issues in parallel, or the user wants to review before any fix lands.
• Fix + PR — additionally root-cause, fix in src/, add the unit test, keep the fixture, and carry it through PR (per "On a confirmed bug" + the merge steps).

Only skip the question when the user already said which (e.g. "issueを立てるまでを ゴールに" = issue-only, or "直して" / "fix and PR" = fix). When in doubt, ASK — a wrong assumption here either wastes a fix that collides with a parallel agent, or stops short of a fix the user wanted.

Core principles

1. Many-people-hit beats niche. Prioritize the resources/configs a large fraction of CDK users deploy every day — S3 (encryption/lifecycle/CORS/ intelligent-tiering), Lambda (arm64/env/tracing/reserved-concurrency/ FunctionUrl/logGroup), VPC (subnets/NAT/routes/endpoints), DynamoDB, IAM, API Gateway, ECS/Fargate, RDS, SQS/SNS, CloudFront — over exotic edge cases. A bug in a daily pattern is worth ten niche ones.

2. The two signals ARE the priority — hunt FP and FN above all else. False positives (drift reported that isn't real) and false negatives (real drift missed) are the bug classes this hunt exists to find; they are what actually damages a user's trust in the tool. Prioritize provoking and confirming them over incidental findings (a crash, a read-gap skipped=, cosmetic output) — those are worth noting, but FP/FN are the target. A freshly deployed stack with NO out-of-band change is the cleanest oracle:

   • False positive (FP) — the most user-damaging class. After record, a check MUST be CLEAN. Any declared-tier drift on a clean recorded stack means cdkrd's declared template value normalizes differently from the live value (a normalization / default-folding bug). record snapshots only the UNDECLARED dimension, so a surviving post-record drift is necessarily a declared-dimension FP — exactly the class worth catching. The invariant (see CLAUDE.md / DESIGN.md): a clean, un-mutated deploy must show ZERO [Potential Drift] on a check BEFORE record, too. Every value AWS assigns at creation that the template never declared is an initial/default, not a divergence — so it MUST fold to atDefault. [Potential Drift] is only ever a REAL divergence (a user change, or an AWS out-of-band change AFTER creation like Application Signals adding IAM perms). So check-before-record on a fresh fixture and read the [Potential Drift] list: every entry there is a fold gap = a bug (the FP the check-output note + issue link ask users to report). When a candidate default's status is uncertain, RESOLVE it by verifying what AWS assigns to a fresh minimal config — never leave it surfaced as "conservative", that just ships the bug. A value the user never changed showing on a first check is the bug, not an acceptable state: do NOT rationalize leaving it undeclared as "honest", and shrinking the count from N to "a few" is not a fix — the target is zero. Fold by escalating through the CLAUDE.md fold-strategy decision order, stopping at the first that applies: (1) equality-gated constant (KNOWN_DEFAULTS / KNOWN_DEFAULT_PATHS) — folds the default, surfaces a change away (detection kept); (2) derived default (CONTEXT_DEFAULTS = f(region), ENGINE_DEFAULTS = f(engine), or a value computed from a sibling / declared prop — e.g. an EB MaxSize default derivable from its EnvironmentType) when the default is a deterministic function of the declared inputs rather than a constant (detection still kept) — before calling a default "context-dependent, can't fold", ask "can I DERIVE it?"; (3) value-independent ONLY as a last resort, for a default AWS moves (platform AMI, versioned URL, GA version) or a per-resource identifier / cosmetic value that cannot be pinned or derived (loses detection — acceptable only because it is undeclared, so a user who cares declares it).
   • False negative (FN) / missed detection — record→check→CLEAN does NOT exercise detection. So ALSO mutate a declared, MUTABLE property out of band (the "someone changed it in the console" scenario — Lambda MemorySize/ Timeout, SQS VisibilityTimeout) and assert check DETECTS it (exit 1), then revert restores it, then check is CLEAN. Pick a MUTABLE property: create-only/immutable ones (Subnet AZ, NAT AllocationId) can't drift.

3. Check coverage first. Before building anything, grep the existing fixtures so you hunt in genuinely-uncovered territory:

   grep -rln "Kinesis\|Dashboard\|Secret\|intelligentTiering\|FunctionUrl" tests/integration/*/app.ts
   

   Empty hits = untested = good hunting ground. But a NON-empty hit does NOT mean the type's undeclared-default scenario is covered: an existing fixture/corpus case that DECLARES the suspect property never exercises the undeclared-default fold, so a first-run FP on that property stays latent under apparent coverage (observed on Events::ApiDestination InvocationRateLimitPerSecond — the pre-existing corpus case declared it, hiding the undeclared-300 FP; #615). Before skipping a "covered" type, check whether any existing case leaves the suspect property UNDECLARED — grep the fixture app.ts / the corpus case's declared block for it; if every case declares it, the undeclared-default path is still open hunting ground.

   The single most reliable FP-finder: deploy each priority type in its BAREST possible config — one type at a time — and check immediately (before record). A minimal fixture declares only what CFn REQUIRES (e.g. an RDS DBInstance with just engine / dbInstanceClass / allocatedStorage / master creds — no version, storage, or retention) and so leaves the MOST properties undeclared → it exercises the MOST default-folds, which is exactly where first-run FPs live. A rich fixture (or a corpus case that declares those props) HIDES them. This is the systematic loop that prevents the whole class: minimal deploy → immediate check → fold every [Potential Drift] to zero → move to the next type. Do not settle for one rich fixture per type. And cover the type's COMMON VARIANTS in their minimal form, not just one — a default is frequently a function of a mode / family / engine, so a variant you did not deploy is an unguarded gap. Concretely (a live-found miss, #1477): the RDS folds were all built from an Aurora / DBCluster-centric corpus, so a minimal non-Aurora provisioned DBInstance still first-ran FPs on StorageType (gp2 — folded only for aurora), BackupRetentionPeriod (1 — added to DBCluster but not DBInstance), and undeclared EngineVersion. When a type has engine / mode / family axes (RDS provisioned vs Aurora, and per-engine; ElastiCache redis vs valkey vs memcached; ECS EC2 vs Fargate; a create-only vs mutable form), enumerate the axis and deploy the MINIMAL form of each branch — the corpus being green on ONE variant proves nothing about the others.

4. Probe CC support BEFORE an expensive deploy — skip the CC-gap tail. For a high-cost or slow stateful/niche type (RDS-family, OpenSearch, MSK, Neptune, DocumentDB, Cloud Map, …), first check whether Cloud Control can even READ it — the FP/FN hunt only has traction on CC-readable types (where AWS's live model diverges from the template by normalization). If the type's CC read throws UnsupportedActionException, every resource comes back skipped=N (surfaced transparently in the info: footer — NOT a false negative): a clean record→ check is hollow and a detect is invisible because the resource was never read. Such a type yields zero FP/FN bugs and has no regression value as a fixture — it is an SDK_OVERRIDES reader candidate (a separate feature task), not a hunt target. So do NOT burn a paid deploy on it. Confirm support first:

   aws cloudformation describe-type --type RESOURCE --type-name AWS::Foo::Bar \
     --query 'ProvisioningType'   # FULLY_MUTABLE/IMMUTABLE = provisionable; then probe READ:
   # if you have a live instance, `cloudcontrol get-resource` — UnsupportedActionException = CC-gap
   

   (Confirmed CC-gap this way: ServiceDiscovery HttpNamespace+Service, DocumentDB DBCluster/DBInstance, AppSync ApiKey/GraphQLSchema — all SDK_OVERRIDES candidates, not hunt targets.)

   The INVERSE is prime hunting ground: an SDK_OVERRIDES reader / SDK_SUPPLEMENTS entry that EXISTS but has zero corpus cases and zero fixtures was added from a live FN report without ever exercising the barest first-run path — deploy its minimal form first (the 2026-07-12 hunt's RedshiftServerless Workgroup trio, #1489, came from exactly this audit; ACM Certificate / ELBv2 TrustStore / DAX / MediaConvert / SageMaker EndpointConfig / ClientVPN remain unexercised).

   A read handler being present is NOT enough — also check the primaryIdentifier ARITY. A type can have a CC read handler yet still be silently skipped with a ValidationException (a DIFFERENT read-gap class than UnsupportedActionException) when its primaryIdentifier is COMPOSITE (more than one segment) but its CFn physical id is only the CHILD segment — Cloud Control GetResource then rejects the bare id. This is a CC_IDENTIFIER_ADAPTERS fix (derive the parent|child / child|parent composite from the resolved declared Ref), NOT an SDK_OVERRIDES one. Probe it offline before deploying:

   aws cloudformation describe-type --type RESOURCE --type-name AWS::Foo::Bar \
     --query 'Schema' --output text | python3 -c "import json,sys; s=json.load(sys.stdin); print(s['primaryIdentifier'])"
   

   primaryIdentifier length > 1, the type is CC-read-able, it is NOT already in CC_IDENTIFIER_ADAPTERS / SDK_OVERRIDES, and the CFn Ref returns only the child segment → a likely declared-read gap worth a (cheap) deploy to confirm the exact composite order (the order is unreliable to guess — verify live, e.g. with aws cloudcontrol get-resource). Confirmed this way: Logs SubscriptionFilter (FilterName|LogGroupName, PR #344). But weight by GENERATION: registry-era types are overwhelmingly NATURAL composites (their CFn physical id is already the seg1|seg2 join, so CC reads them as-is — no adapter needed). The 2026-07-14 ccpi-hunt deployed 7 uncovered composite-pi types in one cheap stack (ServiceCatalog PortfolioPrincipalAssociation + TagOptionAssociation, AppRegistry AttributeGroupAssociation, aoss AccessPolicy, EC2 SecurityGroupVpcAssociation, Lex BotVersion + BotAlias) and ALL read clean — zero skipped. The gap class lives in LEGACY types that kept a bare-segment physical id when registry-migrated (the existing ~40 adapters are all that class); a one-stack association-pack probe is still worth it for new suspects, but expect "no gap" as the common outcome.

5. Predict FP classes from the fold allowlists, then audit them OFFLINE before any paid deploy. The per-type fold tables in src/normalize/noise.ts ARE the inventory of FP classes already found — and most are CURATED, KNOWN-INCOMPLETE allowlists (CASE_INSENSITIVE_PATHS, VERSION_PREFIX_PATHS, UNORDERED_ARRAY_PROPS / UNORDERED_OBJECT_ARRAY_PROPS / UNORDERED_NESTED_OBJECT_ARRAY_PATHS, RATE_EXPRESSION_PATHS, EPOCH_HOUR_PATHS, TRAILING_DOT_PATHS). Each lists only the 1–2 types someone already hit; any OTHER type sharing that semantic divergence is an unguarded gap. So you can PREDICT where FPs hide instead of deploying blind:

   • The recurring FP-generating axes (AWS live value ≡ declared value but ≢ structurally): set-like array reorder (DNS RecordSet values, Cognito URL/OAuth lists, WAF sets, SG rules), partial→concrete version (*Version/EngineVersion/ KafkaVersion a service expands), case-insensitive enum (*Type/*Protocol/ *Status), trailing/format normalization (FQDN dot, ARN :*, rate(), epoch), object↔JSON-string shape (a Definition/Content/policy declared as object, read back as string). Suspect any prop named *Version/*Type/*Protocol/ *Status/trailing-Name(FQDN)/*Arn/Schedule*/map-type/order-insensitive array.
   • Audit the gap OFFLINE first (free). For each candidate, read the allowlist to see what's covered, then grep tests/corpus/*.json: compare resource.declared vs liveRaw for the prop. If a recorded live read EXHIBITS the divergence and expected is clean → the trigger is already covered+guarded (corpus-replay proves it), skip it. If no corpus case exercises the trigger (e.g. a RecordSet case with only ONE value never tests multi-value reorder), or the service can't even produce the divergence (MSK rejects a partial version → declared==live, NO risk) → that determination is the deliverable. Only deploy the genuine, reproducible gaps. This ruled out a whole class and ~10 wasteful deploys in the PR #303 hunt — fan out parallel read-only agents (one per class) to do the audit. The fix for a confirmed gap is usually a one-line allowlist addition + the unit test + corpus case.

6. Parallelize, but cap at 3–4 stacks. Independent stacks (unique names) can deploy concurrently as background tasks, but more is not better — it makes logs and teardown hard to follow. VPC/NAT (~3 min) pace a wave; most others ~1–2 min.

Workflow

1. Worktree + build

Per CLAUDE.md, never work in the main checkout: git worktree add .worktrees/<name> -b wt-<name> main → mise trust .worktrees/<name>/.mise.toml → pnpm install → vp run build (the CLI runs from dist/).

2. Scaffold fixtures + ARM the cleanup gate

Add fixtures under tests/integration/<name>/ — mirror an existing one (app.ts + cdk.json + package.json + verify.sh). A clean-FP verify.sh is: deploy → record --yes → check --fail MUST exit 0. Run npm install then cdk synth for all fixtures in parallel FIRST (cheap, catches TS errors before any paid deploy).

Before deploying, record every stack you are about to deploy into the sentinel — this arms the cleanup gate. SCOPE it to THIS session so a parallel agent's live stacks never mix into a shared owner file (see the owner-scoping note below):

# this session's own owner (env does not persist across tool calls — re-prefix each command)
CDKRD_BUGHUNT_OWNER="session-${CLAUDE_CODE_SESSION_ID:-$$}" \
  .claude/skills/hunt-bugs/bughunt-track.sh add CdkRealDriftIntegS3Rich CdkRealDriftIntegVpcCommon ...

Run every later bughunt-track.sh verify / clear with the SAME CDKRD_BUGHUNT_OWNER="session-${CLAUDE_CODE_SESSION_ID:-$$}" prefix, so you clear only YOUR own pending set. (The deploy-autoarm-gate hook also arms a per-session autoarm-<session> token on any deploy as a backstop — keyed by the SAME CLAUDE_CODE_SESSION_ID — so the merge/commit block is per-session either way; the explicit add gives the clearer gate message and the per-stack list.) Tag every fixture cdkrd:ephemeral=1 (Tags.of(app).add('cdkrd:ephemeral','1') in app.ts) so the generic tag net in sweep-orphans.sh catches any resource type it has no per-type rule for.

3. Deploy (parallel, capped) + check

Run the verify.sh set in parallel (≤3–4). Each verify.sh MUST have a cleanup trap that runs delstack cdk -a cdk.out -r "$REGION" -f -y (NOT cdk destroy — see CLAUDE.md) on EXIT, so even a failed run deletes its stack. Triage every result: that is not CLEAN, and scan the info: footer: a skipped= on a COMMON type is a read-gap many users hit (an SDK-override candidate); an unresolved= points at declared values whose intrinsics cdkrd couldn't resolve.

4. Test detection (the FN half)

For at least one common type, mutate a declared MUTABLE property out of band and assert check detects → revert → check CLEAN → live value restored (lambda-rich/verify-detect.sh is the reference).

When your FP fix ADDS a KNOWN_DEFAULTS fold for a MUTABLE prop AWS assigns, live-test the REVERT of that value too — not just detection. Mutate the folded prop to a NON-default (it must re-surface, proving the equality-gate still detects an out-of-band change), then revert and confirm the live value actually returns to the default. Some providers IGNORE an omitted property on update, so the default remove revert is a SILENT no-op — Cloud Control reports SUCCESS yet the live value persists (observed on Transfer UpdateServer / SecurityPolicyName #597, IAM MaxSessionDuration, Lambda Alias Description, Cognito AllowClassicFlow). The fix is to add ${resourceType}\0${path} to REVERT_SET_DEFAULT_PATHS (src/revert/plan.ts) so revert writes the KNOWN_DEFAULTS default EXPLICITLY and converges — otherwise you ship a revert that claims success but leaves the value unchanged.

The revert-no-op class is NON-UNIFORM — a dedicated-toggle-API does NOT imply a no-op; live-prove EACH candidate, never predict from the API shape. It is tempting to assume that a property changed only by a dedicated sub-API (Enable/DisableRule, Increase/DecreaseStreamRetentionPeriod, PutBackupPolicy, SetQueueAttributes) must no-op on an omitted remove — but the Cloud Control HANDLER often RECONCILES the full desired state and resets the property to its default when omitted. Live-proven 2026-07-13 (#1571): of four dedicated-toggle siblings, only Kinesis::Stream RetentionPeriodHours actually no-oped (CC leaves it unchanged) and needed the RSDP entry; Events::Rule State, SQS::Queue VisibilityTimeout, EFS::FileSystem BackupPolicy all CONVERGED via the bare remove (the CC handler reset them). So a cheap combined fixture (several folded toggles in one stack) that mutates each out of band and asserts the LIVE value after revert is the only reliable test — the API shape is not a predictor. The revert-toggle-converge fixture is the reference (one fixed case + converge-via-remove controls). Batch 2 (2026-07-14, revconv-hunt fixture): ECR::Repository ImageTagMutability no-oped (independently found+fixed the same day as #1580/#1581 — check upstream before pushing a same-table fix); Lambda TracingConfig, SQS DelaySeconds, KMS Key Enabled all converged via bare remove — again ~1-in-4, unpredictable from the API shape. Batch 3 (2026-07-14, revconv2-hunt fixture): ECR::Repository ImageScanningConfiguration no-oped (the ImageTagMutability sibling — same partial-update contract); DDB PointInTimeRecoverySpecification, S3 VersioningConfiguration (the handler actually SUSPENDS on omitted — S3 can never return to never-versioned), LogGroup RetentionInDays, EventBus LogConfig, Kinesis StreamModeDetails (ON_DEMAND→PROVISIONED via remove) all converged. The no-op contract is non-uniform even WITHIN a type: ECR's RepositoryPolicyText / LifecyclePolicy removes DO converge (both policies live-deleted), so the ECR gap is exactly the two scan/mutability scalars — prove per-property, not per-type. Also excluded from any in-run revert probe by AWS-side rate limits (not cdkrd bugs): DDB TTL (1 change/h), EFS ThroughputMode (1 change/24h); Kinesis stream-mode allows exactly 2 switches/24h — enough for one mutate + one revert, none left for a retry. Batch 4 (2026-07-14, revconv3-hunt fixture, #1613): SQS SqsManagedSseEnabled (its 4 scalar siblings converge — non-uniform WITHIN SQS again), SFN LoggingConfiguration, ApiGateway RestApi DisableExecuteApiEndpoint, Cognito UserPoolClient RefreshTokenValidity all no-oped; Athena WorkGroup State, DDB DeletionProtectionEnabled, Scheduler Schedule State converged via the bare remove — ~1-in-2 this round. Excluded as SERVER-SIDE IRREVERSIBLE (not a convergence probe): SSM Parameter Tier (AWS cannot downgrade Advanced→Standard). Batch 5 (2026-07-14, revconv4-hunt fixture, #1619): ECS Cluster ClusterSettings, ApiGateway RestApi ApiKeySourceType, Glue Crawler SchemaChangePolicy no-oped (RSDP entries converge them); CW Alarm TreatMissingData, AppSync IntrospectionConfig, Scheduler ScheduleExpressionTimezone, Pipes DesiredState converged — ~1-in-2 again. And a THIRD class appeared: CloudWatch CompositeAlarm ActionsEnabled no-ops even an EXPLICIT add /ActionsEnabled true CC patch (SUCCESS reported, value unchanged), so an RSDP set-default CANNOT converge it — the fix is an SDK_PROP_WRITERS entry driving the dedicated Enable/DisableAlarmActions API. Probe that class for free before writing the fix: aws cloudcontrol update-resource --patch-document '[{"op":"add","path":"/X","value":<default>}]' against a CLI-created resource answers "does the explicit write converge?" with no stack. Also found: CodeBuild Project + MediaConvert Queue detect fine but are read-only ("type not revertable yet") — when a probe target is such a type, restore it OUT OF BAND before revert or the fixture can never converge to zero (#1623). Batch 6 (2026-07-14, barest4/ccpi hunt): RUM AppMonitor CustomEvents no-oped (silent keep), and ServiceCatalog TagOption Active surfaced a FOURTH flavor — the handler REJECTS the bare remove outright (InvalidRequest: Active and new value cannot both be null), a hard error instead of a silent no-op; both fixed by RSDP entries. Batch 7 (2026-07-14, revconv5-hunt fixture): Lambda RecursiveLoop + RuntimeManagementConfig, HTTP-API DisableExecuteApiEndpoint, SES ConfigurationSet SendingOptions/ReputationOptions, KinesisVideo DataRetentionInHours, CloudTrail EventSelectors, and S3 PublicAccessBlockConfiguration ALL converged via the bare remove (0-in-8 no-op — the class has streaks; keep probing anyway, batches 4-5 were ~1-in-2). The batch's real payoff was the DETECTION side — see the all-boolean-object off-flip gotcha. Batch 8 (2026-07-15, misspack/lattice2/attach2 hunt, probed as piggybacks on that hunt's NEW folds rather than a dedicated fixture): VpcLattice ResourceConfiguration AllowAssociationToSharableServiceNetwork, Backup RestoreTestingPlan StartWindowHours AND ScheduleExpressionTimezone (same handler, both proven individually), and EC2 TransitGatewayAttachment Options ALL no-oped — 4-in-4 for this batch's new folds (streaks run hot too); every one converged via an explicit CC add patch → plain RSDP entries (#1639/#1640/#1642). Piggyback the convergence probe on every NEW KNOWN_DEFAULTS fold a hunt ships (mutate → revert → re-read) — it is ~1-in-3 to need an RSDP entry, and the probe is nearly free while the stack is still up.

5. Harvest the live read into the golden corpus (EVERY round — bug or not)

This is the asset a hunt leaves behind even when it finds no bug. Every live read you just paid for is a real normalize→classify pipeline input; capturing it as a golden-corpus case turns this one-time deploy into a permanent offline regression that runs in plain vp run test (no AWS) forever — tests/corpus/*.json is replayed by tests/corpus-replay.test.ts, which re-runs classifyResource on the recorded inputs and asserts the findings reproduce exactly (R63). A future normalization change that would silently re-introduce an FP/FN on this resource then fails a unit test instead of waiting for the next paid hunt.

So while a tracked stack is still deployed, record the corpus by setting CDKRD_CORPUS_DIR on a check (it writes one sanitized case per readable resource — account ids are stripped at record time):

CDKRD_CORPUS_DIR=/tmp/corpus-<name> node "$ROOT/dist/cli.js" check "$STACK" --region "$REGION"

Record on the FRESH deploy BEFORE record (no baseline) so the case captures the full classification — the atDefault/undeclared folding, not a baseline-snapshotted clean. Then promote the cases that add coverage into tests/corpus/:

• Each file is named AWS__<Service>__<Type>.<LogicalId>.json. Copy in the cases for types not already present — ls tests/corpus/ | grep <Type> first. Genuinely-new resource types are the win; skip near-duplicates of types already covered (VPC/subnet/route-table boilerplate a fixture drags along is usually already represented — don't flood the corpus with it). Also check the exact FILENAME: generic CDK logical ids (VpcpublicSubnet1Subnet…) collide across fixtures, and a same-name cp silently OVERWRITES the existing case — rename the new one with a distinct suffix (e.g. AWS__EC2__Subnet.Ipv6AttachHunt.json) when the id already exists (2026-07-12 hunt clobbered one this way).
• A promoted case whose expected pins an OPEN issue's wrong behavior WILL churn: parallel agents fix filed issues within hours, so after any rebase re-RUN classify over the promoted cases and regenerate expected (a throwaway env-gated test file that rewrites c.expected from classifyResource output beats hand-editing; delete it before commit). Three peer fixes landed mid-hunt on 2026-07-12 alone.
• Run vp run test and confirm the new corpus-replay cases pass. Commit the new corpus JSONs in the SAME PR as the fixture (and the fix, if any). An intended behavior change updates a case's expected in the same diff, making the semantic change reviewable.

The *-rich fixtures are exactly the rich configs worth pinning this way, so a clean round still ships growing regression coverage — see the "A clean result IS a result" gotcha.

5.5 First-run-noise sweep (shrink [Not Recorded] via KNOWN_DEFAULTS)

After promoting new corpus, run the offline first-run-noise sweep — the newly harvested cases are exactly the fresh data it mines:

bash scripts/measure-noise.sh

It replays classify over tests/corpus/*.json and ranks every undeclared (type, path), flagging the constant-looking ones as CANDIDATEs to promote into KNOWN_DEFAULTS (top-level) / KNOWN_DEFAULT_PATHS (nested) in src/normalize/noise.ts. This matters because the CFn schema annotates a default on only ~1% of properties (see scripts/measure-schema-defaults.mjs and docs/ARCHITECTURE.md § 6), so these hand tables — not the schema — are what keeps a first run's [Not Recorded] inventory small. Promote a candidate only when its value is a genuine CONSTANT service default (not a per-resource id/ARN/name/AZ/window the heuristic may over-flag); the fold is equality-gated, so a correct promotion can never hide a real change, and a recorded value that later moves off the default still surfaces. Add the entries + a noise-and-strip test in the SAME PR. This is a quality/noise pass, not a bug — skip it on a round that ships no new corpus.

6. On a confirmed bug: file an issue, then fix it — with a unit test (mandatory)

Always file a GitHub issue for every confirmed bug (gh issue create), even when you fix it in the same session — every bug becomes a tracked, claimable unit, so nothing is silently lost and parallel agents/sessions don't duplicate it. An issue-only hunt round files the issue and stops there (the fix comes later); a fix-in-session round still files the issue, then closes it from the PR (Closes #<n>). The issue body carries the real repro (live model / commands) so the later fixer has the evidence.

When you then WORK an issue — this hunt's own or one already filed — run /work-issues and follow it for the collision-safe start: its §0 screens the issue's comments for untrusted/malware content (first-pass, then defer to the maintainer; never access/run an attachment) and its §4 claims the issue with a gh issue comment BEFORE you edit. Do NOT re-implement those steps here — the /work-issues skill is the single source of truth, so this stays correct when it changes.

Then fix it:

1. Root-cause it in src/ (normalize / diff-classify / read-router / overrides / intrinsic-resolver / report — wherever the divergence-from-reality lives).
2. Fix it in the worktree.
3. Add a unit test that fails without the fix and passes with it. This is mandatory, not optional — a bug found by integ MUST leave behind a unit test that pins the corrected behavior, so the regression can never come back silently (integ alone is too slow/expensive to be the only guard). Re-run vp run build + vp run test.
4. Re-run the live repro with the fixed binary to confirm the real-AWS behavior is now correct.
5. Keep the fixture as a committed regression integ under tests/integration/<name>/, in the SAME PR as the fix — never defer the integ.
6. If the bug is a CLASS, prove it's closed for EVERY affected type — don't stop at the one resource you happened to hit. Most real bugs here are not specific to the type that surfaced them: they live in shared code keyed on a schema flag or a normalizer applied to many types (e.g. #252 — a property that is BOTH write-only and create-only was re-included into a Cloud Control patch and rejected; found on ElastiCache, but RDS / DynamoDB / EC2 / Redshift / S3 / EFS … all have such properties). When the root cause generalizes:
   • Map the blast radius. Enumerate which other types/properties share the trigger — e.g. aws cloudformation describe-type --type RESOURCE --type-name <T> --query Schema and compute the relevant intersection across common types. Name them in the PR so the coverage is visible.
   • Add a DATA-DRIVEN invariant test, not just a per-type one. A hand-built single-type unit test proves the symptom is gone for ONE shape; it does not prove no oversight elsewhere. Drive the test from the golden corpus's REAL schemas: load every tests/corpus/*.json (via reviveSchema), reproduce the trigger for each, and assert the invariant holds for ALL of them. The corpus already spans ~17 real types, and the test self-extends as the corpus grows — a far stronger guard than enumerating types by hand. (tests/revert-plan.test.ts create-only invariant over all real corpus schemas is the reference.)
   • Confirm it fails without the fix and passes with it, like any regression test — then you have proof the whole class is closed, not just one instance.
   • This pairs with step 5's corpus harvest: harvesting rich cases during hunts is what makes the corpus a strong enough substrate to drive these invariants.

7. Cleanup — non-negotiable (see below), then ship

Run /sweep-resources — the shared cleanup phase: it deletes every tracked stack with delstack, sweeps the stack-external orphans (IAM roles, log groups, RETAIN resources, tagged-any-type), verifies SWEEP CLEAN, and releases the gate (bughunt-track.sh verify → clear, incl. this session's autoarm-<session> owner). Then /check + /check-docs markers → commit → push → /verify-pr → gh pr create.

8. Merge + remove the worktree

Take it all the way to merged — do not leave a green PR hanging:

1. gh pr merge <#> --squash --delete-branch (the remote branch). If CI is down for billing, --admin after confirming the local gates passed.
2. Remove the worktree — a hunt always creates one (.worktrees/<name>), and a left-behind worktree is the silent residue of this flow. From the MAIN checkout: git worktree remove .worktrees/<name> (add --force if it refuses on leftover build artifacts), then git branch -D wt-<name> if the branch lingers, and git worktree prune. Confirm with git worktree list — only the main checkout should remain. (Mirror of CLAUDE.md's integrate-then-remove rule.)

9. Record what you learned

For any recurring surprise (a whole class of latent bug, a verification gotcha, a methodology improvement), encode the durable lesson into THIS skill file — a committed principle/gotcha survives across machines and sessions, whereas an auto-memory is per-terminal and invisible to the next hunter. Fold the new lesson into the relevant ## Core principles / ## Gotchas entry (with the issue/PR number as evidence), and PR it. Reserve memory for genuinely session-transient notes; anything future-hunter-relevant belongs in the skill.

Cleanup is non-negotiable (gate-enforced)

Forgetting to delete bug-hunt stacks is the one unacceptable outcome, so it is enforced structurally rather than by discipline:

• bughunt-track.sh add <stacks...> writes the deployed stack names to the gitignored sentinel .markgate-bughunt-pending.
• The bughunt-clean-gate PreToolUse hook (.claude/hooks/bughunt-clean-gate.sh) blocks git commit, gh pr create, and gh pr merge while that sentinel is non-empty — so you physically cannot land the fix PR (or any commit) until the bug-hunt stacks are deleted and verified gone.
• bughunt-track.sh verify asserts each tracked stack is GONE from CloudFormation AND sweep-orphans.sh reports SWEEP CLEAN, and on success STAMPS the verified pending set; bughunt-track.sh clear empties the sentinel (releasing the gate) and REFUSES without a stamp matching the current pending set — "verify passed first" is enforced structurally, not by shell plumbing (a piped verify | tail && clear once chained a clear onto a FAILED verify because the pipeline's exit was tail's). Run verify and clear as separate, un-piped commands from the SAME directory (the owner key is cwd-derived — a cwd that drifted back to the main checkout arms/clears the WRONG owner).
• Owner scoping — set CDKRD_BUGHUNT_OWNER="session-$CLAUDE_CODE_SESSION_ID". When CDKRD_BUGHUNT_OWNER is UNSET, the tracker derives the owner from the main-tree root (--git-common-dir), so two sessions both running add from the main checkout write into ONE shared owner file (#1409). Then a clear — which empties the WHOLE owner file — would drop a PEER's still-pending stacks, releasing the gate while their live AWS resources remain. Setting a per-session owner gives each session its own .d/session-<id> file, so your clear can never touch a peer's. If you DID share the default owner with a peer: NEVER clear it while it lists another session's stacks — release only your own session's token (CDKRD_BUGHUNT_OWNER="autoarm-$CLAUDE_CODE_SESSION_ID" ... clear, or CDKRD_BUGHUNT_FORCE_CLEAR=1 if the only remaining sweep orphan is provably a peer's), and merge from a worktree cwd (the gate scopes commit/merge blocks by the committing command's worktree-toplevel owner + your autoarm-<session> token, not the shared main-root owner).

delstack only deletes stack MEMBERS. sweep-orphans.sh catches the stack-EXTERNAL orphans teardown leaves behind — auto-created /aws/lambda/* log groups (notably from S3 autoDeleteObjects custom-resource Lambdas), RETAIN stateful resources, Secrets in recovery, KMS keys pending deletion. Do NOT delete the sentinel by hand to bypass the gate.

Gotchas (learned the hard way — keep current)

• record hides undeclared FPs. A record→check→CLEAN fixture only proves the DECLARED dimension is FP-free; undeclared mis-classification is snapshotted away. To probe it, check BEFORE record and read the atDefault/unresolved/[Not Recorded] breakdown with --verbose.
• An FN detect-test needs a record between the clean first check and the out-of-band mutation. Without a baseline, the mutated value surfaces only as [Potential Drift] and check --fail still exits 0 — the test false-fails on the exit-code assert even though detection worked (hit live on the MediaConvert Queue pause, #1526). Sequence: deploy → first check CLEAN → record --yes → mutate → check --fail MUST exit 1 (confirmed drift) → restore → CLEAN.
• An UNDECLARED-atDefault prop's OOB change is only caught via "appeared since record", which needs the resource complete — and a DECLARED write-only prop (secret/password/key) used to break that. record snapshots only undeclared NON-default values; a prop at its AWS default folds atDefault and is NOT recorded, so a later OOB change to it is caught ONLY by the R62 "appeared since record" mechanism — which fires ONLY when the resource is complete. A resource that DECLARES a write-only property (RDS MasterUserPassword, any secret/token/key) surfaced a write-only readGap that (pre-#1582) marked it NOT complete, silently disabling that detection — so mutating an undeclared-atDefault prop on it read [Not Recorded], check --fail exited 0, and it looked like a fold bug when it was a completeness gap. When an undeclared-prop FN detect-test on a resource that declares a secret/password unexpectedly MISSES: (1) don't assume a fold/classify bug — check whether the pure classify surfaces it offline (it did here) and whether the resource is complete (a readGap=N in the info: footer is the tell); (2) a MUTABLE prop that AWS applies as an ONLINE modify (RDS CopyTagsToSnapshot/MonitoringInterval, no reboot) keeps the instance available, so aws … wait …-available returns before the change propagates — poll describe (AND cloudcontrol get-resource, a different surface) until the value flips before asserting. The write-only-readGap→incomplete FN was live-found this way (#1582).
• A harvested corpus case can embed a credential-shaped physical id that git-secrets rightly blocks at commit. An AWS::IAM::AccessKey case's physicalId IS a real AKIA… id (the corpus recorder strips account ids, not access-key ids). Sanitize before committing: replace every occurrence with AWS's documented example id AKIAIOSFODNN7EXAMPLE (consistent replace keeps the case self-consistent; corpus-replay only needs internal equality) and re-run corpus-replay (#1526 PR).
• An off-flip FN candidate is real for a STANDALONE-boolean pin OR an ALL-BOOLEAN-object pin — mixed object/array pins are not swallowed. isTrivialEmpty drops a bare false/""/[]/{} AND an object whose every leaf is trivial; a true pinned INSIDE an object whose siblings are non-trivial (ReadWriteType: "All", SSEAlgorithm:"AES256", VersionNumber:1) survives the flip — the flipped shape breaks the pin equality and surfaces normally. An offline audit of "unpaired true-pins" (the #1530 hunt) overcounted 9→4 for exactly this reason: before filing, check the pinned true stands ALONE at its path, and drop pins on immutable-revision resources (ECS TaskDefinition) that cannot drift out of band. But the INVERSE trap (hunt 2026-07-14): a whole-object pin with ONLY boolean leaves (SendingOptions: {SendingEnabled: true}, the 4-flag S3 PublicAccessBlockConfiguration) flips ALL-FALSE when every toggle is disabled, and the all-false object IS trivially empty — swallowed before the pin gate exactly like a standalone bool (the GuardDuty DataSources shape, #1092). The class is mechanically enumerable: scan KNOWN_DEFAULTS for object pins whose leaves are all booleans, then live-probe each for (a) OOB mutability (AmazonMQ EncryptionOptions = create-only, S3 AccessPoint PABC = no mutate API, VpcLattice SharingConfig = update API can't flip it, EMRServerless monitoring = service rejects the all-false state — all EXCLUDED) and (b) the off-state READ shape (all-false object = fixable via MEANINGFUL_WHEN_OFF; ABSENT-from-read like a deleted S3 PAB config = the separate vanished-undeclared- default limitation, not fixable by a pin gate). The 2026-07-14 sweep confirmed + fixed S3 Bucket PABC (check stayed CLEAN while a bucket was opened to public access — the most security-critical FN found to date), SES ConfigurationSet SendingOptions/ReputationOptions (live end-to-end), and SES EmailIdentity DkimAttributes/FeedbackAttributes (CC-read-shape proven).
• A corpus promotion can PIN a live FP as expected — eyeball declared-tier findings before promoting. The #1507 hunt promoted three RDS cases whose expected carried the mixed-case-name declared FP (CdkrdHunt-Mixed-CPG vs cdkrdhunt-mixed-cpg), so corpus-replay asserted the WRONG behavior until #1531. A declared finding on a name/identifier path where declared and live differ only by case (or another pure normalization) is a red flag: that is a bug to file, not an expectation to record.
• A KNOWN_DEFAULTS pin containing an ARRAY must carry the exact live element shape. matchesKnownDefault is subset-tolerant for OBJECT keys but strict deep-equality for arrays — trivially-empty element sub-keys (CidrListAliases: [], CommonName: "") must be IN the pin or it never matches (hit on the Lightsail Networking default-firewall pin, #1533). Copy the array verbatim from the live read.
• When a reader's physical-id-shape assumption breaks (name vs ARN), grep the SAME service family's sibling readers for the identical assumption — in both directions. readSageMakerEndpointConfig passed the ARN physical id as the bare name and ValidationExceptioned on every read (#1527) while its sibling readSageMakerMonitoringSchedule had already fixed exactly that (#1523) — the fix existed one function away. A physical-id shape is per-type but the MISTAKE is per-family; audit siblings before shipping a one-type fix.
• A revert bug's fix belongs in the route the plan ACTUALLY takes — check SDK_WRITERS[type] FIRST. A type with an SDK writer never sends the CC patch, so a fix in the CC augmentation path (augmentCcItemOps strips) is dead code for it. The #1568 Glue capacity-echo failure was first "fixed" in the CC layer before discovering writeGlueJob existed — the real bug was the writer's non-WorkerType branch re-sending BOTH GetJob capacity echoes. Corollary: a writer's unit-test mock must mirror the REAL read echo shape (the old test's GetJob mock omitted the dual MaxCapacity+AllocatedCapacity echo, so the always-failing branch looked green); copy the mock model from a live read, not from the declared template.
• Post-update echo materialization is its own FP class — probe it with a neutral update. A clean FIRST-run check proves nothing about the post-update echo surface: Glue normalizes sizing on EVERY UpdateJob (including a CFn stack update), so undeclared WorkerType/NumberOfWorkers materialized out of nowhere after an update that never mentioned them (#1569) — and the pair is irremovable (a remove revert is a structural no-op). After the first-run check, run ANY update against the resource (a service update API call or a trivial template change) and re-check: every newly materialized undeclared field is a latent FP a real user hits on their second deploy. Fold it with the same decision order (the sizing pair joined the existing value-independent MaxCapacity trio). The CHEAP wide-sweep form (2026-07-14): one combined barest stack of many common types whose app.ts threads a -c rev=2 context into a stack tag + per-resource descriptions — deploy, first check, redeploy with rev=2, re-check; the second-deploy-echo/-echo2 fixtures swept 27 common types this way (no new echo found; Lambda materialized an empty VpcConfig husk post-update — correctly dropped). Two lessons: (a) a barest multi-type echo stack doubles as a first-run FP probe and that is where it actually paid (the Kinesis StreamModeDetails FP below); (b) check --fail exits 0 on baseline-less potential drift, so a first-check assert MUST grep "Potential Drift" the output — trusting the exit code let a real FP print INTEG OK.
• A barest PROVISIONED variant hides behind its richer/other-mode siblings. The 2026-07-14 hunt's only first-run FP: a Kinesis stream declaring ONLY ShardCount reads back StreamModeDetails={"StreamMode":"PROVISIONED"} undeclared — every existing fixture either declared StreamModeDetails or was ON_DEMAND (#1487's fold is the exact inverse: ON_DEMAND materializes ShardCount). When a type has a mode axis, deploy the barest form of EACH mode and check which sibling props the OTHER mode's fold assumed declared.
• The variant axis extends to UNION-TYPED config blocks and to defaults the variant FLIPS on a sibling. Two live instances (variants2-hunt, 2026-07-14): (a) Firehose's destination union — every fixture was ExtendedS3, so a barest HttpEndpointDestinationConfiguration first-ran 7 nested-echo FPs; each destination variant carries its OWN default family (HTTP's S3BackupMode default is FailedDataOnly while ExtendedS3's is Disabled — do NOT copy a sibling variant's constants, read them from the live echo). (b) EFS One Zone (AvailabilityZoneName declared) FLIPS the BackupPolicy default to ENABLED — the Regional constant DISABLED pin missed it; the fix is a tier-2 derived fold gated on the declared variant marker, with the Regional constant as the fall-through. When probing a variant, expect it to change defaults on OTHER properties, not just add its own. Fixture foot-gun: Firehose VALIDATES the endpoint URL shape at create — a .invalid-TLD placeholder is REJECTED (Invalid Url), so use https://example.com/<path> (reserved, resolvable, never actually called by a DirectPut stream with no producers).
• A write-only re-include can be a side-effectful WRITE, not a keep-alive — watch for revert-manufactured drift. Reverting an unrelated prop (TracingConfig) on a ZipFile Lambda re-included Code.ZipFile per the CC read-modify-write contract, which the handler executed as UpdateFunctionCode: the zip re-packaged (non-deterministically), live CodeSha256 moved off the recorded baseline, and the revert's own convergence check reported a drift the revert itself created (permanent until re-record). Fix class: WRITEONLY_REINCLUDE_SKIP in revert/plan.ts (live-proven 2026-07-14). When a revert leaves a synthetic read signal (CodeSha256/ScriptSha256/bundle sha) "remaining", suspect the patch's own re-include before calling it an AWS bug.
• Sibling-ATTACHMENT echo materialization is the post-update class's twin — deploy the ATTACHED shape, not just the barest parent. A parent deployed ALONE can hide FPs that only materialize when a sibling attaches to it: a barest ClientVpnEndpoint (the existing clientvpn-barest fixture) reads back neither VpcId nor SecurityGroupIds, but the moment an in-stack ClientVpnTargetNetworkAssociation lands, BOTH materialize (the subnet's VPC + its default SG) → 2 first-run FPs invisible under apparent coverage (#1574, 2026-07-13). When a type has attachment-style siblings (association / attachment / registration / membership resources), deploy the parent WITH one attached and first-check that shape too. Fold with the decision order — the association echo is usually tier-2 derivable from the declared sibling (here: SubnetId → in-stack Subnet.VpcId, plus the shared DEFAULT_SG_LIST gate). Related probe mechanics: ClientVPN authorization-rule revoke is ASYNC (the rule lists as revoking for ~30s+ — an FN probe that checks too early false-passes; poll until the rule is GONE before asserting detection), and a rogue SG applied to an endpoint can't be deleted until the swap-back propagates to the association ENIs (DependencyViolation — retry with a wait loop).
• Immutable props can't drift. Don't treat an unresolved/unverifiable create-only property (Subnet AvailabilityZone via Fn::Select(Fn::GetAZs), NAT AllocationId via Fn::GetAtt EIP) as a bug — it's correctly classified. And don't "fix" it by resolving Fn::GetAZs: AZ ordering differs from DescribeAvailabilityZones, risking an FP, for zero detection benefit.
• set -e aborts inline multi-step bash. An interactive shell with set -e stops a one-off inline script right after a check --fail that exits 1. Put the detect→revert→re-check sequence in a standalone verify.sh (with explicit || fail), or guard with set +e.
• Always npm install + cdk synth before deploy — a synth-time TS error is free to catch; a half-failed deploy is not.
• A container-image Lambda fixture MUST build with docker build --provenance=false --sbom=false. Docker 24+ buildkit attaches OCI provenance/SBOM attestation layers by default; Lambda rejects them at CREATE with InvalidImage: UnsupportedImageLayerDetected (the stack rolls back). The failure is NON-DETERMINISTIC (a prior build sometimes slips through), so it reads as flakiness — pin the flags. Also build --platform linux/amd64 unless the function declares Architectures: [arm64] (a barest function defaults to x86_64 and rejects an arm64 image). Push the image to a dedicated ECR repo out of band and pass the registry/repo@sha256:… digest via env — the barest CfnFunction then declares only Code.ImageUri + PackageType: Image + Role, leaving the Image-variant defaults (Architectures, EphemeralStorage, RecursiveLoop, LoggingConfig, RuntimeManagementConfig) undeclared to probe the fold (lambda-container-barest is the reference; container Lambdas were a whole uncovered variant axis until #1572).
• Deploy-time API validation differences across engine/mode variants are FINDINGS, not mere fixture bugs. A minimal variant deploy that FAILS validation is telling you the variant's defaults differ (valkey RGs default AutomaticFailoverEnabled=true — a 1-node group is rejected — and demand an explicit TransitEncryptionEnabled, both unlike redis; observed 2026-07-12). Record the difference in the fixture comment (it documents the axis) and declare the minimum to proceed — the surviving undeclared surface is still the probe.
• Raw-API acceptance ≠ CloudFormation reachability — probe the CFn HANDLER before concluding a case-echo FP risk. Several CC handlers add CLIENT-side validation the raw service API does not have: elasticache create-user and memorydb create-parameter-group both ACCEPT a mixed-case identifier (storing it lowercased — the FP trigger), yet the CFn/CC handlers REJECT the same input (InvalidRequest: must contain only lowercase…), making the FP unreachable via CloudFormation — no allowlist entry needed (2026-07-13, #1539 determinations). The cheap sequence: probe the raw API by CLI create+delete first (it answers "does the service lowercase?"), but only a CFn DEPLOY proves reachability; a handler rejection is itself the determination (record it in the fixture comment). The inverse also held: Redshift's and Batch's handlers pass mixed case through, and both FP'd. Cheaper still: aws cloudcontrol create-resource exercises the SAME CFn/CC handler with NO stack and NO fixture — a rejection comes back as a FAILED progress event in seconds, and an acceptance gives you the stored echo via get-resource before you delete-resource. The 2026-07-14 hunt determined the whole remaining MemoryDB family (SubnetGroup/User/ACL all reject: "must contain only lowercase") and Cassandra Keyspace (accepts AND preserves case — no FP either way) for zero deploys; reserve the paid CFn fixture for the types the handler lets THROUGH (Redshift::Cluster ClusterIdentifier, #1589). Tag the probe resource cdkrd:ephemeral=1 in its desired state and delete it immediately.
• An undeclared-revert "proof" is void if the CDK L2 declares the leaf. Before claiming a revert no-op / convergence proof for an UNDECLARED property, read the DEPLOYED template: mutating a value the L2 silently declared (RDS CopyTagsToSnapshot: true) produces NO divergence (live == declared), so the revert never plans it and the "proof" proves nothing — the fixture-side twin of the "corpus that declares the target leaf can't stand in" lesson. Live-proof each REVERT_SET_DEFAULT sibling individually with a template that genuinely omits it (#1541: BackupRetentionPeriod proven; CopyTagsToSnapshot stays unproven for exactly this reason).
• An in-stack scalable target is a cheap real-drift generator. A ScalableTarget scheduled action (min/max below the declared capacity) makes App Auto Scaling clamp the resource within minutes of deploy — producing a REAL capacity divergence with no out-of-band CLI call. That accident exposed the WarmThroughput creation-echo FP (#1538: warm throughput echoes CREATION capacity and never follows a scale-in, so a derived fold gated only on the CURRENT live sibling FPs after any scale-in). Pattern to reuse: derived folds for creation-echo values must ALSO gate against the DECLARED-derived value, and autoscaling-governed fixtures probe that class for free.
• example.com / .test / .example are AWS-RESERVED for Route53 hosted zones (InvalidDomainNameException on create). A Route53 fixture must use a non-reserved placeholder domain (e.g. cdkrd-fphunt-x9z7q.com.) — a public hosted zone for a domain you don't own still creates fine (it just isn't authoritative).
• Not every type is revertable — the FN half may stop at detection. Route53 RecordSet, for instance, has no SDK writer (revert says "type not revertable yet"), so a detect→revert→clean cycle can't complete. Prove the FN by mutating the declared value out of band (e.g. aws route53 change-resource-record-sets) and asserting check --fail exits 1, then restore it manually; note the revert gap as a future SDK_WRITERS candidate rather than treating it as a regression.
• Read the revert's convergence REPORT text, not just the live value — the report layer has its own bug class. A revert whose target converges perfectly can still print a false NOT reverted: …MasterUserPassword — the default-value write was a no-op for the write-only RE-INCLUDE op every password-declaring resource carries (the CC read-modify-write contract): a write-only path re-reads as readGap with no live value on either side, so a persistence check built on deepEqual(pre, post) is vacuously true (#1594, live-hit on an aurora-pg Sv2 revert 2026-07-14). When a revert probe passes on the live value, ALSO grep its output for NOT reverted: / could not be confirmed on paths you never drifted — an unverifiable (readGap) path must never drive a "value persists" verdict, and fixtures that only assert check --fail exit codes ride right past this class.
• A declared+undeclared FP PAIR with the SAME value at sibling paths = a stored KEY SYNONYM — canonicalize the declared side, after probing the echo via Cloud Control. GuardDuty stores a Filter's short condition keys as their long twins (declared Criterion.severity.Gte: 4 reads back GreaterThanOrEqual: 4), so one declared short key produced BOTH a declared "removed" finding and an undeclared "appeared" finding with equal values (#1612). The tell is that value-equal pair. Probe the echo shape for free before fixing: aws cloudcontrol create-resource with the short keys, get-resource back — the CC read echoed ONLY the long forms (the raw GetFilter returns both, but cdkrd reads via CC). Fix = a declared-side key canonicalization scoped to the criterion map (canonicalizeGuardDutyCriterionKeys).
• A curated per-name creation-status map re-breaks every time AWS launches an OFF-by-default feature — that is its designed failure mode; the fix is one line. GuardDuty Detector Features folds via GUARDDUTY_FEATURE_CREATION_STATUS (classify.ts), which errs toward VISIBILITY: a new opt-in protection AWS ships DISABLED (AI_PROTECTION, 2026 — #1612, after #1485's AI_ANALYST) surfaces the whole array as a first-run FP until its name is added. When a barest detector FPs on Features, check that map FIRST — do not reach for value-independent (that was reverted once already, #1092: it hid out-of-band disables forever).
• CDKRD_CORPUS_DIR exported around a whole verify-detect.sh records EVERY check — the LAST (post-mutation) read wins. A detect/revert script runs 3-4 checks; the corpus case for a mutated resource then pins the MUTATED read, and a later measure-noise sweep flags the mutated value as a bogus CANDIDATE default (hit on Conv3PoolClient RefreshTokenValidity: 60, 2026-07-14). Scope the env to the FIRST (clean) check line only — or, if the mutated case is worth keeping as a detection pin, promote it under the existing .drifted.json naming so its intent is explicit.
• A sweep-orphans.sh fix made in a WORKTREE does not take effect for bughunt-track.sh verify — the tracker resolves the script at the MAIN tree root (--git-common-dir), so a phantom-orphan fix (a new resource_gone arm) authored in the hunt worktree still fails verify against the unpatched main copy, deadlocking the gate the fix exists to release (hit 2026-07-14 on the VPN-family arms). Resolution: temp-copy the patched script over the main checkout's, run verify + clear, then git -C <main> checkout -- tests/integration/sweep-orphans.sh to restore main to HEAD — the committed fix lands permanently at merge. Never force-clear instead.
• The uncovered-type well is nearly dry — most remaining corpus-missing types are dead, closed, or expensive, so audit ALIVENESS before building a fixture. The 2026-07-15 sweep enumerated every corpus-missing type: the bulk are EOL/closed to new customers (QLDB, CodeCommit, MediaStore, Evidently, Pinpoint, Timestream LiveAnalytics, S3 Object Lambda — live create rejects with "available only to existing customers", determined the hard way via a rolled-back deploy), account singletons unsafe to touch (Macie/Inspector/Detective/SecurityLake), or cost-prohibitive (ACMPCA $400/mo, FSx, EKS nodegroups, MWAA). The surviving cheap tail (EMR SecurityConfiguration, SageMaker Pipeline/ModelPackageGroup, Transfer Workflow, Location APIKey, SES DedicatedIpPool, XRay ResourcePolicy, Backup RestoreTestingPlan, VpcLattice AuthPolicy/ALS/ResourceGateway/ResourceConfiguration/ SNVpcAssociation, TGW Attachment, SES CSED) was deployed by misspack-hunt / lattice2-hunt / attach2-hunt — future hunts should pivot to variant/echo/ attachment/notation angles rather than re-mining the missing-type list. Also determined there: same-account Oam::Link is REJECTED ("Cannot create a link to a sink in the same account" — cross-account only, unprobeable solo), and a lowercase-only-name service (VPC Lattice) mints its CFn generated name LOWERCASED (cdkrdhunt0715lattice-sn-<random>), which the exact-case isCfnGeneratedName branches missed until #1639.
• A per-variant fold TABLE row that was MIRRORED from a live-proven sibling is itself unproven — audit the split tables for never-deployed rows. The BY_PROTOCOL / BY_LB_TYPE / BY_TARGET_TYPE-style variant tables are built one live variant at a time, and the untested rows get filled by copying the proven sibling's constants "for symmetry" — which bakes the do-NOT-copy-sibling-constants trap INTO the fold table (ELB_TG_ATTRIBUTE_DEFAULTS_BY_PROTOCOL's UDP/TCP_UDP rows carried TCP's deregistration_delay.connection_termination.enabled: 'false'; AWS's UDP-family default is 'true' → first-run FP, #1664). A barest deploy of each mirrored-row variant is cheap (a TargetGroup needs no LB); grep the split tables for rows whose comment cites a DIFFERENT variant's deploy as evidence.
• An FN detect-probe needs its resource at readGap=0 — a reader-projection readGap silently disables appeared-since-record for the WHOLE resource, and the report masks it as "No baseline yet". The R62 mechanism only fires on snapshot-COMPLETE resources; a declared prop the reader never projects (DLM's shorthand DefaultPolicy, #1665) keeps the resource incomplete forever, so every undeclared OOB change stays [Potential Drift] and check --fail exits 0 — and (pre-#1665) the preamble printed "No baseline yet" right after a successful record, sending you to re-record instead of at the readGap. When a detect probe unexpectedly misses: check the target resource's readGap= in the info: footer FIRST, and either close the gap (project the declared-shaped value from what the API does return, gated so it never emits on other shapes — the #1660 lesson) or probe a readGap-free sibling resource. The readGap-closing fix then needs the SAME live proof pair as any reader fix: clean first run + detection restored.
• A clean result IS a result — but it must still leave an asset. "6 common+rich stacks, zero FPs, detection+revert verified" is a legitimate, valuable outcome. Do NOT manufacture a fix to have something to show. The deliverable of a bug-free round is the committed *-rich fixtures PLUS the golden-corpus cases harvested from their live reads (step 5) — that is how a clean round still grows permanent offline regression coverage instead of evaporating when the stacks are torn down.
• Before salvaging leftover fixtures from an interrupted worktree, check for an already-merged duplicate. A half-finished prior hunt can leave uncommitted fixtures in a stale worktree, and resuming them is tempting — but a PARALLEL session may have already merged the identical dirs under a differently-ordered PR title (this flow's ecr-rich/kinesis-rich/secrets-rich salvage collided with the already-merged #248 "kinesis-rich, secrets-rich, ecr-rich"). Run gh pr list --state merged --search "<type-name>" AND git ls-tree -d --name-only origin/main tests/integration/ | grep <name> for the fixture names FIRST — before any paid re-deploy — and abort if they already exist. A clean abort (remove the worktree; the AWS side was already swept) beats burning a deploy on a duplicate PR that will only conflict.
• Fixture buckets MUST set removalPolicy: DESTROY — the L2 Bucket default is RETAIN, and a rollback/teardown then silently ORPHANS the bucket (a failed variants3-hunt deploy left DELETE_SKIPPED on its bucket, 2026-07-14; the sweep's ephemeral-tag net catches it, but don't rely on that). Same for any L2 stateful default-RETAIN construct in a hunt fixture.
• A service that VALIDATES a role's permissions at create races a grant()-style attached policy — use inlinePolicies in hunt fixtures. Firehose validated glue/s3 access before the separate AWS::IAM::Policy resource attached (the DeliveryStream only depended on the role) and failed create; inline policies are part of the role create, so no dependency plumbing is needed (variants3-hunt, 2026-07-14). Related create-time determinations worth reusing: a table-less Firehose Iceberg destination is REJECTED ("a single default destination table configuration must be provided" — DestinationTableConfigurationList is part of the barest form), and a CFn Glue Iceberg table requires TableType: EXTERNAL_TABLE.
• Working a filed issue → run /work-issues (don't re-implement its rules here). The issues this hunt files get picked up by later parallel sessions that race for the same ones and collide on the same central tables (noise.ts / classify.ts / revert/plan.ts). /work-issues owns the collision-safe start — claim the issue with a gh issue comment before editing, screen untrusted comments, pick file-disjoint lanes — and is the single source of truth so it stays correct as it evolves (see also the "Claim a filed issue before working it" rule in CLAUDE.md).
• Filing an issue attracts malware bait — never run an attachment OR install a package a stranger posts on it. This hunt's deliverable is public issues, and a hostile actor watches new issues/PRs to reply within minutes with a "helpful fix" that is really a way to make you run unvetted code (the maintainer holds AWS credentials — a prime target). The vector varies but the play is identical — seen live from ONE campaign: issue #648 got a *_fix.zip attachment 4 min after filing; PR #655 (the very PR adding this rule) got pip install vulnledger && vulnledger scan . seconds after merge — a fabricated package (no such real tool). Both from author_association: NONE throwaway accounts, with body text parroting the thread's wording and no real root cause. Do NOT download / unpack / pip install / npm i / curl | sh any of it — read only the comment body via gh api repos/<o>/<r>/issues/comments/<id>, and verify any suggested package name by SEARCH, never by installing. On a match, tell the user and (on their say-so) minimizeComment classifier SPAM → delete → block + report the author; prefer a Web-UI manual block over gh api PUT user/blocks/<user> (404s without the user scope — do not gh auth refresh to widen the token). See CLAUDE.md's "Never download … untrusted third-party content" rule.