aws-infra-migration

name	aws-infra-migration
description	Drive a fenced, one-writer-at-a-time migration of a Terraform root's live state from one CI/repo into another (or into a central Terraform pipeline) with a zero-diff, identity-preserving cutover. Use when moving Terraform roots between repos or CI systems, consolidating infra into central Terraform CI, re-keying remote state, or when the user says "infra migration", "state cutover", "fenced cutover", "move the <X> root", or opens a per-root migration window. Read-mostly: every live state mutation is explicitly gated on a human go.
disable-model-invocation	true

AWS infra migration — fenced one-writer state cutover

Move a Terraform root's live remote state from one CI/repo (the source) to another (the destination) without split-brain, keeping every managed resource's identity so running workloads never notice.

The one invariant: exactly one writer for a given state key at every instant. Source and destination use different backend keys, so the DynamoDB lock table locks them independently — there is no shared lock. "Copy the object and leave the old workflow armed" is split-brain. You make one writer, fence the other, and only ever move forward.

Orient first (do not reinvent)

The per-root procedure should be authoritative in the destination infra repo, conventionally at docs/runbooks/fenced-cutover.md, with a transfer-manifest.template.md and filled exemplars under docs/runbooks/manifests/. Read the runbook and the most recent exemplar; copy the exemplar's shape. This skill is the operator's wrapper around that runbook — it does not replace it. If they disagree, the runbook wins (and fix this skill).

If the destination repo has no such runbook yet, the first migration should produce one; a per-root manifest is the only rollback path you will have.

Actor split (who can do what)

Action	Actor
Edit .tf / backend key, open PRs, drive CI, fence source, fill manifest	dev (you)
Backend state object ops (head-object / copy-object / snapshot) + hand-apply bootstrap/ + policy-simulation	admin credentials (a privileged SSO/role profile) — the dev role cannot touch backend state
Go / no-go at every gate and before every state mutation	the human owner

Check the admin profile is live before you rely on it (aws sts get-caller-identity --profile <admin>); if expired, ask the owner to re-login. Never admin-merge (gh pr merge --admin) past branch protection on repos that enforce reviews or commit signatures.

Per-root loop

For each root, in order (PAUSE for the owner's go before any live mutation — fence toggle, bootstrap apply, copy-object, destination apply/merge, source retire):

1. Pick the next root by correctness, not effort. Prefer clean single-root canaries first (no cross-state ownership, no secret-bearing state, no live prod-user dependency, doesn't own the backend bucket). Leave the cluster root and the backend-bucket root for last.

2. Pre-gates (below) — all green or STOP.

3. Fence the source apply path (see Fence strategy).

4. Drain in-flight source runs; confirm the source key's lock is released.

5. Snapshot (admin): source VersionId + ETag + length (head-object); lineage + serial + instance count; config-dir + .terraform.lock.hcl hashes. Record every value in the manifest — the version IDs are the only rollback path.

   Never read state with a bare terraform state pull | jq. An unfiltered jq is the identity filter: it prints the whole state document, including every plaintext sensitive attribute, to your terminal and into any CI log or shell history that captures it. Stream the object and project only the three scalars you need — this form is safe on every root, so use it as the default rather than remembering to switch on the secret-bearing ones:

   aws s3 cp "s3://<bucket>/<key>" - --profile <admin> \
     | jq '{lineage, serial, instances: ([.resources[].instances | length] | add)}'
   

6. Copy the exact version (admin, after go). Gate 3 established the destination key was absent, but that was a check at a point in time — between the gate and the copy, anything with write access could create it. Make the copy itself refuse to overwrite rather than trusting the earlier check:

   aws s3api copy-object --profile <admin> \
     --copy-source "<bucket>/<srckey>?versionId=<V>" \
     --bucket <bucket> --key "<destkey>" \
     --if-none-match '*'
   

   --if-none-match '*' fails the request with PreconditionFailed if the destination key already exists, closing the time-of-check/time-of-use window. Record the VersionId the copy returns — post-apply the destination has a newer version, and rollback needs to name the exact one it is restoring from. Then verify dest lineage/serial/count/ETag EQUAL the recorded values. Any mismatch = STOP.

7. Add infra/<root>/ to the destination (fresh worktree): byte-identical except the backend ; keep . Open the PR → destination plan (read the actual plan comment — "No changes" — not just a green check; plan exits 0 on a diff). Merge → sole-writer apply . Verify post-apply serial unchanged.

Pre-gates (the make-or-break part)

Gate 1 — CI policy for the root's services, hand-applied FIRST. The destination CI plan/apply roles need least-privilege permissions for exactly this root's services before its first plan/apply.

• Derive the action set from what the provider ACTUALLY calls, not from CRUD. Enumerated CRUD is not enough — the AWS provider makes implicit read calls on every refresh that fail the first plan with AccessDenied if missing. The reliable way to enumerate them is a TF_LOG=trace plan under admin creds, then grep the signed requests for the action names.

  Treat the specific call-level claims below as observations, not as a spec. Which implicit reads a refresh makes is provider-implementation behavior and changes across provider versions — the AWS authorization references (action names, ARN forms) are stable and authoritative, but "this resource's Read also calls X" is only true of the provider version you traced. Record the provider version alongside the grant in the manifest, and re-trace rather than re-use the list after a provider major bump. Three classes bite repeatedly:

  • Tag reads — but check HOW the service returns tags first. If the root's provider.tf sets default_tags (or a resource sets tags), the mutating side (TagResource/UntagResource) is needed on the apply role for any tag change. For the READ side it depends on the service:

    • Services whose Read/Describe does not return tags make a separate tag-read call on every refresh via the transparent-tagging interceptor. Miss it and the first plan AccessDenieds. Grant the read to both roles.

    • Services whose Read/Describe returns tags inline make no separate call — e.g. secretsmanager:DescribeSecret returns Tags in its response, so default_tags adds no read action. Do not invent a GetTags-style grant it will never use.

    • The tag-read VERB SPELLING is per-service — a plausible guess grants nothing, because IAM silently no-ops an action name that doesn't exist for that service. These are all real, all different, and all mean "read this resource's tags":

      Service	Tag-read action
      CloudWatch (alarms)	ListTagsForResource
      CloudWatch Logs	ListTagsForResource

Gate 2 — source 0/0/0 vs live, at a pinned freeze SHA (source main tip). The migration zero-diff gate can't tell expected-state from un-captured drift, so reconcile any drift into source first. If the source drift-detection workflow is fenced/disabled, either temporarily re-enable it, dispatch a single read-only plan for this root, confirm "No changes", and re-disable; or run terraform plan locally under admin creds. Record the run/evidence.

Gate 3 — destination key absent (head-object <destkey> → 404). This is a point-in-time check, not a guarantee: pair it with --if-none-match '*' on the copy (step 6) so the write itself is what enforces non-overwrite.

Gate 4 — backend bucket versioning is Enabled.

aws s3api get-bucket-versioning --bucket <bucket> --profile <admin> --query Status

Every rollback instruction in this document assumes a prior object version can be retrieved by ID. If versioning is Suspended or absent, there is no rollback path and the whole procedure is a one-way door — stop and fix that before touching any state. Confirm the value; do not infer it from the bucket having been created by a module that usually enables it.

Hard rules / gotchas

• Verify the root's ACTUAL resources — trust neither its name nor its README. Grep the .tf for ^resource/^data; a root can be misnamed (e.g. one called "Athena" that declares only Glue catalog resources) and its README can describe an aspirational design that was never built. Scope the IAM grant to what the code declares, confirmed against live resource names. Also check for a for_each/module that inflates one ^resource block into many live instances (e.g. a "VPC" root that is one aws_vpc_endpoint block plus a terraform-aws-modules/vpc/aws module = subnets/NAT/IGW/EIP/route-tables/flow-log role) — the instance count and the policy surface follow the live state, not the block count.

• Scope to exact resource ARNs when the naming namespace is shared. If the root owns only some resources under a name prefix and other, unmanaged resources share that prefix, a <prefix>/* grant over-reaches (worst case: the apply role could delete an unmanaged prod secret/bucket/queue). Enumerate the exact ARNs the root declares (list them live and diff against the .tf), not the prefix. Shape of the problem: a Secrets Manager root declares some secrets under per-environment prefixes, but those same prefixes also hold datastore and third-party secrets it does not manage — so the grant must enumerate one exact secret:<name>-* ARN per declared secret (the -* matches Secrets Manager's random 6-char suffix), and a policy-simulation asserting that an unmanaged same-prefix secret is DENIED is what proves no leak. When the root does own a whole dedicated namespace with nothing else in it, a <prefix>/* grant is fine.

• Implicit reads with irregular ARN forms + multi-resource auth. Some services' refresh reads authorize against ARN shapes you won't guess, and against several resource types at once — get either wrong and the first plan AccessDenieds.

  • Identity Store ARNs are irregular. identitystore:GetGroupId/DescribeGroup authorize against Identitystore = arn:aws:identitystore::<account>:identitystore/<id> (empty region, carries the account) OR Group = arn:aws:identitystore:::group/<group-id> (no region, no account, no store-id in the path). Grant both forms; the "standard" arn:aws:identitystore:<region>:<account>:… shape denies.

Fence strategy (whole-migration decision)

Decide once with the owner:

• Global disable (fewer reviewers): keep the source apply workflow disabled for the entire migration → every remaining root is already fenced, so each cutover skips its own fence PR. Bulk-retire all moved .tf in one PR at the end. Cost: source can apply no infra until the migration ends, and moved roots get no drift monitoring meanwhile (consider a minimal drift cron in the destination).
• Per-root code fences (apply stays available): re-enable source apply and fence each root with its own scoped code PR. More reviewer overhead; restores source apply sooner.

Rollback

Only possible if Gate 4 passed and the manifest captured version IDs.

Rollback re-arms a writer against a key the destination now owns, which is the same split-brain hazard as the forward move — so it gets the same discipline, step for step, not just a "fence the destination first" gesture:

1. Fence the destination apply path.
2. Drain in-flight destination runs and confirm the destination key's lock is released — a rollback that copies over a key mid-apply is the worst outcome available.
3. Snapshot the destination as it stands now (head-object → VersionId/ETag; streamed lineage/serial/count per step 5). You are about to overwrite it and will want the ability to undo the undo.
4. Select the exact version to restore. Normally the latest destination version, never the stale source object — the destination has applied since the cutover and its state has moved on. Name the version ID explicitly from the manifest; do not rely on "current".
5. Copy it into place, then verify lineage/serial/count against what you expect.
6. Re-enable exactly one writer.

Note that step 5 is the one place --if-none-match '*' is wrong — you are deliberately overwriting an existing key. That is precisely why steps 1–3 have to be real: the guard that protects the forward copy is unavailable here, so draining and snapshotting are the only things standing between a rollback and a lost state lineage.

Inventory every write path before declaring the destination fenced. Disabling the one obvious apply workflow is not enough if a drift-detection cron, a scheduled plan-and-apply, a reusable workflow called from elsewhere, or an operator's local credentials can also write that key. Enumerate them; fence all of them.