data-lake-infrastructure

name

data-lake-infrastructure

description

Generates Terraform for data lake infrastructure from an infrastructure manifest. Produces reusable modules, per-environment application stacks, tfvars, and backend configs for S3 buckets, Glue jobs, IAM, KMS, and supporting services. Use when provisioning or modifying data lake infrastructure, adding buckets or Glue jobs, wiring a new environment, or aligning infrastructure with generated pipeline configs.

Data Lake Infrastructure Skill

Skill Metadata

Field	Value
Name	Data Lake Infrastructure
Version	1.0.0
Category	Infrastructure as Code / Data Platform
Tools Required	execute_bash (python3, terraform CLI), fs_write, read_file
Generator	scripts/generate_infrastructure.py
Validator	scripts/validate_terraform.sh
Primary Input	config/infra_manifest.json
Secondary Input	Generated pipeline configs in output/configs/
Output	Terraform modules, application stacks, per-env tfvars and backend configs

Trigger Phrases

• "Generate Terraform for the data lake"
• "Provision the data lake buckets"
• "Create the Glue job infrastructure"
• "Add a new environment (sandbox/stage/prod)"
• "Wire up the SFTP / Transfer Family users"
• "Update infrastructure to match the pipeline configs"
• "Generate the CI/CD pipeline stack"

---

Relationship to data-lake-config

These two skills are complementary and share one contract.

                    config/infra_manifest.json
                    (bucket names, layer topology, paths)
                       │                    │
        reads ─────────┘                    └───────── reads
          │                                              │
          ▼                                              ▼
  data-lake-config                          data-lake-infrastructure
  STTM ──► job configs + DDL                manifest ──► Terraform
       (what runs)                              (what it runs on)

Critical rule: neither skill may hardcode a bucket name or S3 path. Both derive them from the manifest. This is what prevents Glue jobs from writing to buckets Terraform never created.

Before generating, confirm the manifest is the same file the config generator used. If output/configs/ exists, cross-check that the bucket names in those configs match the ones this skill resolves.

---

Input Contract: infra_manifest.json

The manifest is the authoritative source for naming and topology. This skill reads it and must not invent bucket names.

Blocks consumed

Block	Used for
naming.variables	Terraform variable defaults and tfvars values
naming.patterns	locals blocks that construct resource names
naming.layers	for_each set for the data lake bucket module
naming.layer_aliases	Mapping medallion layer to physical bucket/database
path_conventions	Bucket prefix structure, lifecycle rule scoping
service_buckets	Non-data buckets (glue scripts, temp, metrics)

Resolved example

With the current manifest (org=deckers, data_env=dev, domain=consumer, account_id=123456789012, region=us-west-2):

deckers-dev-consumer-datalake-raw
deckers-dev-consumer-datalake-staging
deckers-dev-consumer-datalake-processed
deckers-dev-123456789012-us-west-2-glue

Manifest blocks this skill needs but the manifest does not yet define

Do not silently invent these. If a block is absent, use the documented default and report the assumption to the user:

Missing block	Purpose	Default if absent
lifecycle	Per-layer S3 storage class transitions and expiry	No lifecycle rules; warn
security	KMS key policy, TLS enforcement, versioning	Secure baseline (see standards.md)
environments	Account ID per environment	Single account from naming.variables; warn

Adding these to the manifest is preferred over adding them as Terraform variables, so both skills continue to read one contract.

---

Terraform Standards

Normative. Every generated artifact must satisfy all seven, and any change to scripts/generate_infrastructure.py must preserve them.

#	Standard	Enforced by
TS-1	Use reusable modules for repeated resource patterns (S3, Lambda, Glue, etc.)	6 modules generated; no resource declared inline in a stack
TS-2	Store modules in /modules/{service-name}/	Kebab-case directory per service, each with variables.tf and outputs.tf
TS-3	Store application stacks in /stacks/application/{stack-name}/	Stack root holds main.tf, locals.tf, variables.tf, outputs.tf, versions.tf
TS-4	Each stack has its own environments/ folder with per-env .tfvars and .tfbackend.hcl	One pair per environment; never shared across stacks
TS-5	Use templatefile() for configurations needing variable interpolation	IAM policy documents rendered from policies/*.json.tftpl
TS-6	Use for_each and dynamic blocks where appropriate	for_each for buckets, databases, jobs, inline policies; dynamic for lifecycle rules and their nested transition and expiration blocks
TS-7	All resources must be tagged using a shared tags module	modules/tags/ output passed to every taggable resource; no inline tag maps

Two consequences that are easy to get wrong:

• Modules never contain environment-specific values. A literal account ID, region, or bucket name inside a module violates TS-2 — modules declare variables, environments/{env}.tfvars supplies them.
• locals.tf is the only place names are constructed, mirroring naming.patterns. Building a name elsewhere creates a second source of truth.

aws_glue_catalog_database has no tags argument in provider 6.x, so glue-catalog carries layer and managed_by in parameters instead. A provider limitation, not an exemption — apply TS-7 wherever it is supported.

standards.md has worked HCL for TS-5 to TS-7; templates.md shows representative idioms. Neither is a complete catalog — generate_infrastructure.py is.

---

Repository Layout

Generated Terraform follows this structure, satisfying TS-2, TS-3, and TS-4:

output/infrastructure/
├── modules/
│   ├── s3-bucket/
│   ├── glue-job/
│   ├── glue-catalog/
│   ├── glue-workflow/
│   ├── cicd-pipeline/
│   ├── iam-role/
│   ├── kms-key/
│   ├── dynamodb-table/
│   ├── sns-topic/
│   └── tags/
└── stacks/
    ├── bootstrap/
    │   └── tfstate/            ← local backend; run once per account/region
    │       ├── main.tf
    │       ├── variables.tf
    │       ├── outputs.tf
    │       ├── versions.tf
    │       ├── README.md
    │       └── environments/
    │           ├── dev.tfvars
    │           ├── sandbox.tfvars
    │           ├── stage.tfvars
    │           └── prod.tfvars
    └── application/
        └── datalake/           ← S3 backend, created by the bootstrap stack
            ├── main.tf
            ├── variables.tf
            ├── locals.tf
            ├── outputs.tf
            ├── versions.tf
            ├── policies/
            │   └── glue-s3.json.tftpl
            ├── buildspecs/
            │   ├── validate.yml.tftpl
            │   ├── plan.yml.tftpl
            │   └── apply.yml.tftpl
            └── environments/
                ├── dev.tfvars
                ├── dev.tfbackend.hcl
                ├── sandbox.tfvars
                ├── sandbox.tfbackend.hcl
                ├── stage.tfvars
                ├── stage.tfbackend.hcl
                ├── prod.tfvars
                └── prod.tfbackend.hcl

Two stacks, applied in order. The bootstrap stack creates the state bucket the application stack's backend points at, so it cannot itself use that backend — it runs on a local backend and has its own KMS key. See Bootstrap Stack below.

Module directories are kebab-case, one per service. Every stack carries its own environments/ folder — tfvars are never shared across stacks.

---

AWS Service Versions

Pinned versions. Do not vary these per environment — an environment that runs a different Glue or provider version is no longer a valid pre-production check of the one above it.

ID	Service	Version / Config	Where it lands
SV-1	Terraform	>= 1.8.0	versions.tf → required_version
SV-2	AWS Provider	~> 6.0	versions.tf → required_providers.aws.version
SV-3	Glue	5.0 (PySpark 3)	glue-job module default; actual value from each config's execution.glue_version
SV-4	Lambda Runtime	python3.11	Not yet generated
SV-5	CodeBuild Image	aws/codebuild/amazonlinux2-x86_64-standard:5.0	cicd-pipeline module, from cicd.codebuild_image
SV-6	Transfer Family Security Policy	TransferSecurityPolicy-2024-01	Not yet generated

Enforced today: SV-1, SV-2, SV-3, SV-5. The generator warns when a pipeline config carries a Glue version other than 5.0.

Forward-looking: SV-4 and SV-6. There is no modules/lambda/ or modules/transfer/. The values are recorded so whoever adds them does not choose arbitrarily — do not present them as satisfied.

Glue 5.0 is Spark 3.5 / Python 3.11 with native Iceberg, so the merge and SCD2 paths need no --datalake-formats argument or bundled JARs. Fix version mismatches in the config generator, never by overriding in Terraform — the config stays the source of truth for job runtime.

---

Environments

Environment	Purpose	Account	Tier
dev	Development / testing	Non-prod	non-prod
sandbox	Integration testing	Separate	non-prod
stage	Pre-production	Separate	pre-prod
prod	Production	Separate	prod

Every stack generates all four environment file pairs regardless of which --env is requested, so adding an environment never requires a stack change.

Three of the four environments occupy their own AWS account, so each needs its own account_id from the manifest environments block:

"environments": {
  "dev":     {"account_id": "…", "region": "…", "tier": "non-prod"},
  "sandbox": {"account_id": "…", "region": "…", "tier": "non-prod"},
  "stage":   {"account_id": "…", "region": "…", "tier": "pre-prod"},
  "prod":    {"account_id": "…", "region": "…", "tier": "prod"}
}

Failure modes:

• Placeholder account IDs. An unreplaced REPLACE_WITH_* targets nothing real. Generator warns per environment, validator per file. Never plan or apply against one.
• Absent environments block. Everything falls back to naming.variables.account_id, silently pointing prod.tfvars at the dev account. Always reported for that reason.
• prod scope. Say so plainly and confirm before writing.

tier is emitted as a tag and is currently informational — the intended hook for stricter pre-prod and prod guardrails (deletion protection, longer KMS windows).

---

Manifest Schema

Formal contract for config/infra_manifest.json — required and optional blocks, field types, validation rules, defaults for absent blocks, and change impact across both skills:

#[[file:.kiro/skills/data-lake-infrastructure/manifest-schema.md]]

Standards Reference

Terraform idioms, AWS service versions, environment definitions, tagging, and the security baseline:

#[[file:.kiro/skills/data-lake-infrastructure/standards.md]]

Module and Stack Templates

Concrete file templates for each module, the stack root, tfvars, and backend configs:

#[[file:.kiro/skills/data-lake-infrastructure/templates.md]]

---

Workflow

Full Generation (Happy Path)

For full Terraform generation, invoke the generator script:

python3 scripts/generate_infrastructure.py \
  --manifest config/infra_manifest.json \
  --output ./output/infrastructure \
  --env {env}

This handles Steps 1-6 automatically. The agent should:

1. Confirm the target environment with the user
2. Run the script
3. Report the resolved bucket names, assumptions, and lifecycle summary it prints
4. Run the validator (below)
5. Flag the security-relevant resources it lists

The script will exit non-zero and generate nothing if the manifest is invalid or if a pipeline config references a bucket that is neither provisioned nor declared in external_sources. Do not work around this — fix the manifest or regenerate the configs.

Bootstrap Stack — apply first, once per account and region

The application stack's backend "s3" cannot create its own state bucket, so the bootstrap stack uses a local backend and its own KMS key — no dependency either way.

cd output/infrastructure/stacks/bootstrap/tfstate
terraform init                                    # local backend, no -backend-config
terraform apply -var-file=environments/dev.tfvars

cd ../../application/datalake                     # only works after the above
terraform init -backend-config=environments/dev.tfbackend.hcl

• prevent_destroy = true on the bucket — state loss is unrecoverable.
• Versioning is mandatory, not advisory — the only recovery path for corrupt state.
• Its own state is local: commit it, or terraform init -migrate-state after the first apply. Committing is fine here — a bucket and a key, no secrets.

Bucket and alias names come from the manifest tfstate block and match the bucket value in every generated tfbackend.hcl. Do not edit one without the other.

Validate-Only Mode

To check the manifest and cross-check pipeline configs without writing files:

python3 scripts/generate_infrastructure.py --validate-only

Validation

After generating, always run:

./scripts/validate_terraform.sh

This runs terraform fmt -check -recursive, terraform init -backend=false, and terraform validate, plus structural checks. Useful flags:

Flag	Effect
--fix	Apply formatting instead of only checking
--structural	Skip terraform, run structural checks only
--stack PATH	Validate a stack other than the default

If terraform is not installed the script falls back to structural checks and states plainly that terraform-level validation did not run. Report that limitation rather than presenting the output as fully verified.

Iterative Modifications

For targeted changes the agent edits files directly rather than regenerating — see Conversational Patterns below. Anything driven by the manifest (bucket names, layers, lifecycle, environments) must go through the manifest and a regeneration, never a hand-edit of generated Terraform.

Step Reference

Step 1: LOAD & VALIDATE MANIFEST
   |  Resolve all names; report missing optional blocks
   v
Step 2: CROSS-CHECK AGAINST PIPELINE CONFIGS
   |  Confirm bucket names match output/configs/ (if present)
   v
Step 3: DETERMINE RESOURCE INVENTORY
   |  Which modules and how many instances of each
   v
Step 4: GENERATE MODULES
   |  One per service; no environment values inside
   v
Step 5: GENERATE APPLICATION STACK
   |  main.tf, locals.tf, variables.tf, versions.tf, outputs.tf
   v
Step 6: GENERATE PER-ENVIRONMENT FILES
   |  {env}.tfvars and {env}.tfbackend.hcl for each environment
   v
Step 7: VALIDATE
   |  ./scripts/validate_terraform.sh
   v
Step 8: REPORT
      Summarize resources, flag anything requiring review before apply

Steps 1-6 are implemented in scripts/generate_infrastructure.py; Step 7 in scripts/validate_terraform.sh. The sections below document what each step does so the behaviour can be reviewed and extended — they are not instructions to reimplement it inline.

---

Step 1: Load and Validate Manifest

Implemented by the Manifest class and validate_manifest() in scripts/generate_infrastructure.py.

What it does:

• Loads the manifest and resolves data_env / account_env from --env
• Pulls account_id and region from the environments block when present
• Applies documented defaults for absent optional blocks, recording each as an assumption to report
• Validates required keys, that every layer_aliases value exists in naming.layers, that every pattern placeholder resolves, and that lifecycle transitions are ordered and do not conflict with expirations
• Flags placeholder account IDs and any security value that weakens the baseline

Validation rules and their severities are specified in manifest-schema.md. Errors stop generation; warnings and assumptions are reported and generation continues.

The script reports resolved names before generating anything:

Resolved from config/infra_manifest.json (env=dev):
  raw        -> deckers-dev-consumer-datalake-raw
  staging    -> deckers-dev-consumer-datalake-staging
  processed  -> deckers-dev-consumer-datalake-processed
  glue       -> deckers-dev-123456789012-us-west-2-glue

Assumptions (manifest blocks absent):
  lifecycle    -> none generated
  environments -> all environments use account 123456789012

---

Step 2: Cross-Check Against Pipeline Configs

Implemented by cross_check() in scripts/generate_infrastructure.py.

Every S3 URI in output/configs/**/*.json is extracted and its bucket compared against what the stack provisions. Each referenced bucket falls into one of three categories:

Category	Meaning	Behaviour
Provisioned	Resolved from naming.patterns	Expected
External	Declared in manifest external_sources	Granted read-only IAM access
Orphan	Neither of the above	Hard error — generation stops

Orphans mean jobs would read from or write to a bucket that does not exist. The script exits non-zero and writes nothing.

Also reported:

• Unused — provisioned but never referenced. A warning; may be intentional.
• Glue version drift — a warning if any config's glue_version differs from the 5.0 standard. Reconcile the config generator rather than overriding in Terraform, so the config remains the source of truth for job runtime.

This check is the reason the manifest exists. Skipping or suppressing it reintroduces exactly the drift it prevents.

---

Step 3: Resource Inventory

Derive what to create from the manifest and, where available, the pipeline configs.

Module	Instances	Source of truth
s3-bucket	one per entry in naming.layers, plus one per service_buckets	manifest
kms-key	one per bucket group (data lake, service)	manifest
glue-catalog	one database per layer, named via layer_aliases	manifest
glue-job	one per config in output/configs/{bronze,silver,gold}/	pipeline configs
iam-role	one Glue execution role, plus per-vendor scoped roles if SFTP is in scope	manifest + project context
glue-workflow	one workflow plus triggers when orchestration.enabled	pipeline config dependencies
dynamodb-table	one ETL audit table when observability.audit_table.enabled	manifest
sns-topic	one alert topic when observability.notifications.enabled	manifest
cicd-pipeline	one pipeline plus three CodeBuild projects per application stack when cicd.enabled	manifest cicd
tags	shared module, consumed by every taggable resource	standards

Glue job sizing comes from each config's execution block (worker_type, num_workers, timeout_minutes, retry_count) — do not re-derive it.

Orchestration: the workflow DAG

Job definitions alone do not run. glue-workflow builds the graph from the dependencies array in each pipeline config. Nothing is hand-listed. Satisfies FR-2.

Resource	Count	Derivation
aws_glue_workflow	1	orchestration.workflow_name_pattern
aws_glue_trigger SCHEDULED	1	starts every root job — those with no dependencies
aws_glue_trigger CONDITIONAL	one per distinct dependency set	predicate state = SUCCEEDED on each predecessor

Triggers are grouped, not per-job: jobs with an identical dependency set share one. That turns 10 non-root jobs into 8 triggers today.

SCHEDULED  cron(0 6 * * ? *)  ->  6 Bronze jobs
silver-01..06   after 1 Bronze each  ->  1 Silver each
gold-07         after dim_customer, fact_orders, dim_product
gold-08         after dim_customer, fact_orders, fact_clickstream

build_dag() guards three cases, each a warning: a dependency on a nonexistent job is dropped (it would fail at apply), self-edges are dropped, and jobs unreachable from a root are reported since they would never run. Group names are layer plus stable index, so they do not churn between runs.

Disabled or absent orchestration.enabled means count = 0 — jobs still exist and can be started manually.

CI/CD: deployment pipeline (NFR-3)

One pipeline per application stack, keyed by stack name — NFR-3 requires them separate.

Stage	Action	Notes
Source	CodeStarSourceConnection	DetectChanges = true — webhook, not polling
Validate	CodeBuild	fmt -check → init -backend=false → validate
Plan	CodeBuild	plan -out=tfplan, saved as an output artifact
Approve	Manual approval	dynamic stage, only when require_approval
Apply	CodeBuild	apply tfplan — the approved plan, not a fresh one

• Three CodeBuild projects via for_each over a phase map, so a fourth phase is a map entry.
• Apply consumes the plan artifact; recomputing it would let unreviewed changes land.
• Buildspecs are .tftpl in the stack, rendered by templatefile(), keeping the module generic. Escape shell braces as $${VAR} or templatefile() demands a Terraform variable of that name.
• Terraform is pinned and installed explicitly, so pipeline and local runs match.
• Disabled or absent cicd.enabled creates nothing; deployment stays manual.

Two limits to state plainly rather than let the user discover:

• source.connection_arn is supplied, not created. CodeConnections starts PENDING and needs a console OAuth handshake Terraform cannot perform. While empty, plan fails on Source — the generator warns in every tfvars.
• The CodeBuild apply role is broad (s3:*, glue:*, iam:*, kms:*, dynamodb:*, sns:*, logs:*). It runs apply across the whole stack; narrowing means enumerating every permission. See Known IAM exceptions.

Observability resources

Conditional on the manifest observability block, wired with count so the stack stays valid when absent. Both encrypted with the same CMK as the buckets.

Resource	Manifest key	Purpose
aws_dynamodb_table	observability.audit_table	One row per ETL run: job_id, run_timestamp, records read/written, duration, status
aws_sns_topic	observability.notifications	ETL success and failure alerts
aws_iam_role_policy.glue_observability	either enabled	Scoped dynamodb:PutItem/UpdateItem/GetItem and sns:Publish for the Glue role

Satisfies FR-6 (every run logged, 90-day TTL on expires_at) and NFR-4 (PAY_PER_REQUEST, not provisioned).

• Runtime counterpart: Terraform creates them; the Glue jobs write via write_audit_row() and publish_notification() in scripts/glue_jobs/utils/metrics_writer.py. Names flow from the same manifest block through generate_configs.py, so the job writes to the table this stack provisions.
• Both helpers swallow their own exceptions — an audit or notification failure must never fail a successful ETL run.
• Audit rows complement the S3 metrics JSON: S3 holds the full DQ report, DynamoDB a queryable run log with expiry.
• Email subscriptions need manual confirmation. Terraform reports success before the recipient clicks the link, so an unconfirmed subscription silently receives nothing.

---

Step 4-6: Generation Rules

Implemented by InfrastructureGenerator in scripts/generate_infrastructure.py, which is the source of truth for emitted content. templates.md illustrates the idioms that output follows, for a subset of modules — consult it to learn the patterns, not to predict the file list.

All seven Terraform Standards (TS-1 through TS-7) apply to the generated output. One additional invariant is specific to generation and not covered there:

Object-typed variables receive exactly their declared attributes. Terraform object types are strict — an extra or missing attribute fails validate. When a manifest block carries generator-only fields (such as security.kms.strategy), exclude them from tfvars rather than widening the variable type.

Two manifest blocks require transformation rather than direct pass-through:

• lifecycle — the manifest's per-layer shape is converted into the list-of-rules the s3-bucket module expects, with prefix_rules becoming additional rules scoped by filter.prefix.
• security — split into bucket_security and kms_security stack variables, filtered to the attributes those object types declare.

---

Step 7: Validation

Run the validator and report its results:

./scripts/validate_terraform.sh

It performs, in order:

1. terraform fmt -check -recursive across modules and stacks
2. terraform init -backend=false — module resolution and provider download
3. terraform validate — configuration correctness
4. Structural checks — brace balance, module wiring, undeclared variables, required variables present per environment, backend keys, object-variable attribute matching, hardcoded bucket names, IAM wildcards, placeholder account IDs

All must pass before presenting output as complete. Add --stack stacks/bootstrap/tfstate to validate the bootstrap stack too.

When terraform is not installed only structural checks run. Do not call the output verified — state which checks ran and which did not.

terraform plan needs credentials and reads live state. Do not run it unprompted.

---

Safety

Infrastructure carries materially more risk than config generation. Apply these without exception.

Action	Rule
Generating Terraform files	Proceed freely
terraform fmt, init -backend=false, validate	Proceed freely
terraform plan	Ask first — needs credentials, reads live state
terraform apply	Never run. Present the plan and let the user apply
terraform destroy	Never run under any circumstance
Modifying existing state files	Never
Generating IAM policies	Generate, then explicitly flag for review
Generating KMS key policies	Generate, then explicitly flag for review

• Least privilege by default. Scope IAM to specific ARNs and prefixes. Do not emit Resource: "*" or Action: "s3:*" outside the two roles under Known IAM exceptions, and only with the reason stated to the user.
• Flag every security-relevant resource in the report: IAM roles and policies, KMS keys and policies, bucket policies, SFTP user access scopes.
• Never commit secrets. Credentials belong in Secrets Manager, referenced by ARN. No secret values in tfvars or .tf files.
• Production requires explicit confirmation. When --env prod, say so and confirm before writing.
• Preserve state safety. Backends must specify S3 state with locking; never generate a stack without a backend block.

The validator's wildcard check reads only policies/glue-s3.json.tftpl. Wildcards in inline stack policies — including the CI/CD roles — are not caught automatically. Review those by hand.

---

Error Handling

The generator enforces most of these and exits non-zero. When it does, report the failure and fix the cause — do not bypass the script and write Terraform by hand.

Condition	Response
Manifest not found	Generator exits 1. The manifest is required — this skill cannot invent bucket names.
Manifest missing a required naming.* key	Generator exits 1 naming the absent key.
naming.layers empty	Generator exits 1. At least one layer is required.
layer_aliases value not in naming.layers	Generator exits 1.
Pattern placeholder cannot be resolved	Generator exits 1 naming the placeholder.
Lifecycle transitions unordered, or expiration precedes a transition	Generator exits 1. Objects would be deleted before transitioning.
Bucket referenced by configs, not provisioned, not in external_sources	Generator exits 1 listing the orphans. Fix the manifest or regenerate the configs.
Optional block absent (lifecycle, security, environments)	Proceed with documented default. The assumption is printed — repeat it to the user.
Placeholder account ID (REPLACE_WITH…)	Warning at generation and again per-file in the validator. Never plan or apply against a placeholder.
security value weakens the baseline	Warning, applied but reported. Never silent.
glue_version differs from 5.0	Warning. Reconcile the config generator, not Terraform.
terraform validate fails	Fix and re-validate. Do not present failing Terraform as complete.
terraform not installed	Structural checks only. State explicitly that terraform-level validation did not run.

---

Conversational Patterns

Initial generation

1. Confirm the target environment
2. Run scripts/generate_infrastructure.py --env {env}
3. Report what it printed: resolved bucket names, Glue databases, manifest warnings, assumptions, lifecycle summary, cross-check result, file counts
4. Run ./scripts/validate_terraform.sh and report which checks ran
5. Repeat the security-relevant resource list for review
6. State the plan command but do not run it

Report the script's actual output rather than paraphrasing it. The warnings it emits — placeholder account IDs, weakened security values, Glue version drift — exist to be surfaced, not summarized away.

Modifications

Anything the manifest drives goes through the manifest plus a regeneration — never a hand-edit of generated Terraform, which the next run would overwrite.

Request	Action
"Add a new layer"	Add to naming.layers and naming.layer_aliases, then regenerate both skills. The config generator resolves paths from the same block.
"Add an environment"	Add to the manifest environments block, then regenerate. Produces {env}.tfvars and {env}.tfbackend.hcl; the stack is unchanged.
"Change the bucket naming"	Edit naming.patterns, regenerate both skills. Warn that Terraform does not rename buckets — this creates new ones and orphans the old. Treat as a migration.
"Add a Glue job"	Regenerate pipeline configs first, then run the infrastructure generator. The job set is derived from output/configs/, never hand-listed.
"Add lifecycle policies"	Add or edit the manifest lifecycle block, then regenerate.
"Change the ETL schedule"	Edit orchestration.schedule (AWS cron, six fields), then regenerate.
"Change job dependencies"	Edit the STTM, regenerate configs, then regenerate infrastructure. The DAG is derived from config dependencies — never hand-edit a trigger.
"Disable orchestration"	Set orchestration.enabled false. Jobs remain, triggers are removed.
"Rename the state bucket"	Edit tfstate.bucket_pattern. Warn that this orphans existing state — it is a migration, not a rename.
"Set up the deployment pipeline"	Edit the manifest cicd block, then regenerate. Tell the user the CodeConnections handshake is a console step they must do.
"Add a pipeline for another stack"	Append to cicd.stacks. Each entry gets its own pipeline and its own three CodeBuild projects.
"Skip approval in dev"	Set environments.dev.require_approval false. Warn that apply then runs unattended on every push.
"Change which branch deploys"	Edit cicd.source.branch_per_env, then regenerate.
"Change bucket security"	Edit the manifest block, then regenerate. Any value weakening the baseline is reported as a warning.

Regeneration commands:

# Manifest-driven change affecting both skills
python3 scripts/generate_configs.py --sttm config/sttm_template.xlsx --env dev --output ./output
python3 scripts/generate_infrastructure.py --env dev
./scripts/validate_terraform.sh

Questions to ask rather than assume

• Does each environment use a separate AWS account? (The manifest holds one account_id.)
• Should the Terraform state bucket be created by this stack or does it already exist?
• Is SFTP / Transfer Family in scope, or metadata-only ingestion?
• One KMS key for all buckets, or per-layer keys?

---

Verification Checklist

Run the scripts first — they enforce the mechanical checks and report PASS / FAIL / WARNING themselves:

python3 scripts/generate_infrastructure.py --env <env>
./scripts/validate_terraform.sh
./scripts/validate_terraform.sh --stack stacks/bootstrap/tfstate

Between them they cover module structure (TS-1 to TS-4), Terraform idioms (TS-5 to TS-7), fmt / init / validate, object-variable attribute matching, bucket and database name resolution, config orphans, placeholder account IDs, Glue version drift, and DAG reachability. Do not restate those here — report what the scripts said.

Confirm the following by hand, because no script can:

• Coverage was not overclaimed. SV-4 (Lambda) and SV-6 (Transfer Family) are not generated. FR-1 ingestion, FR-5 QuickSight, FR-6 CloudWatch alarms, and NFR-2 Secrets Manager have no implementation.
• require_approval is true for stage and prod.
• The broad CodeBuild apply role was disclosed to the user, not left implicit.
• An empty cicd.source.connection_arn was surfaced as blocking, not skipped.
• No secret values appear in any generated file.
• Output came from the generator, not hand-written Terraform.
• If terraform was unavailable, that limitation was stated rather than implied.
• terraform apply was not run.

Known IAM exceptions

Most roles are scoped to specific ARNs and prefixes. Two are not, deliberately:

Role	Scope	Why
codebuild apply policy	s3:*, glue:*, iam:*, kms:*, dynamodb:*, sns:*, logs:* on *	Runs terraform apply across the whole stack; narrowing means enumerating every stack permission
codepipeline StartBuilds	codebuild:StartBuild, codebuild:BatchGetBuilds on *	Actions are scoped; only the resource is *, since project ARNs are not known until the module is instantiated

Both are flagged in comments in the generated policy. Tell the user rather than letting them find it during review.