infrastructure-cicd-data-engineering

# Infrastructure CI/CD for Data Engineering > Skill by [ara.so](https://ara.so) — Data Skills collection This project demonstrates practical CI/CD patterns for deploying data infrastructure changes using GitHub Actions, Terraform, and AWS. It uses OpenID Connect (OIDC) for secure, keyless authentication between GitHub Actions and AWS, eliminating the need for long-lived AWS credentials. ## What This Project Does - **Bootstraps infrastructure**: Creates S3 backend for Terraform state and OIDC provider for GitHub Actions - **Automates deployments**: Uses GitHub Actions workflows to plan and apply Terraform changes - **Enforces reviews**: Requires manual approval before production deployments - **Validates code**: Runs Terraform formatting and validation checks on PRs ## Project Structure ``` . ├── terraform/ │ ├── bootstrap/ # Initial setup (S3 backend, OIDC) │ │ └── main.tf │ └── main/ # Main infrastructure definitions │ └── main.tf ├── .github/ │ └── workflows/ │ ├── ci.yml # Format and validation checks │ └── deploy.yml # Deployment workflow └── tear-down.sh # Cleanup script ``` ## Prerequisites 1. **AWS Account** with appropriate permissions 2. **Terraform** installed locally (v1.0+) 3. **GitHub Account** and repository access 4. **AWS CLI** configured with credentials ```bash # Verify Terraform installation terraform version # Verify AWS credentials aws sts get-caller-identity ``` ## Bootstrap Setup ### Step 1: Create S3 Backend and OIDC Provider The bootstrap process creates: - S3 bucket for Terraform state storage - DynamoDB table for state locking - IAM OIDC provider for GitHub Actions - IAM role that GitHub Actions will assume ```bash # Initialize and apply bootstrap configuration terraform -chdir=terraform/bootstrap init terraform -chdir=terraform/bootstrap apply # Capture the outputs terraform -chdir=terraform/bootstrap output ``` **Expected output:** ``` github_actions_role_arn = "arn:aws:iam::123456789012:role/github-actions-role" state_bucket_name = "my-terraform-state-bucket" ``` ### Step 2: Configure GitHub Repository Secrets Create a repository secret named `AWS_ROLE_ARN`: 1. Navigate to: `Settings → Secrets and variables → Actions → New repository secret` 2. Name: `AWS_ROLE_ARN` 3. Value: The ARN output from bootstrap (without quotes) ```bash # Example ARN format (don't include quotes when pasting) arn:aws:iam::123456789012:role/github-actions-role ``` ### Step 3: Create GitHub Environment Set up a production environment with manual approval: 1. Navigate to: `Settings → Environments → New environment` 2. Name: `production` 3. Configure protection rules: - ✅ Required reviewers (minimum 1) - Add yourself or team members as reviewers ## Bootstrap Terraform Configuration **terraform/bootstrap/main.tf** (simplified example): ```hcl terraform { required_version = ">= 1.0" required_providers { aws = { source = "hashicorp/aws" version = "~> 5.0" } } } provider "aws" { region = var.aws_region } # S3 bucket for Terraform state resource "aws_s3_bucket" "terraform_state" { bucket = "${var.project_name}-terraform-state-${var.environment}" tags = { Name = "Terraform State Bucket" Environment = var.environment ManagedBy = "Terraform" } } resource "aws_s3_bucket_versioning" "terraform_state" { bucket = aws_s3_bucket.terraform_state.id versioning_configuration { status = "Enabled" } } # DynamoDB table for state locking resource "aws_dynamodb_table" "terraform_locks" { name = "${var.project_name}-terraform-locks" billing_mode = "PAY_PER_REQUEST" hash_key = "LockID" attribute { name = "LockID" type = "S" } tags = { Name = "Terraform State Lock Table" Environment = var.environment ManagedBy = "Terraform" } } # OIDC provider for GitHub Actions resource "aws_iam_openid_connect_provider" "github_actions" { url = "https://token.actions.githubusercontent.com" client_id_list = [ "sts.amazonaws.com" ] thumbprint_list = [ "6938fd4d98bab03faadb97b34396831e3780aea1" ] } # IAM role for GitHub Actions resource "aws_iam_role" "github_actions" { name = "github-actions-terraform-role" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [ { Effect = "Allow" Principal = { Federated = aws_iam_openid_connect_provider.github_actions.arn } Action = "sts:AssumeRoleWithWebIdentity" Condition = { StringEquals = { "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com" } StringLike = { "token.actions.githubusercontent.com:sub" = "repo:${var.github_org}/${var.github_repo}:*" } } } ] }) } # Attach policies to the role resource "aws_iam_role_policy_attachment" "github_actions_admin" { role = aws_iam_role.github_actions.name policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess" } # Outputs output "github_actions_role_arn" { value = aws_iam_role.github_actions.arn description = "ARN of the IAM role for GitHub Actions" } output "state_bucket_name" { value = aws_s3_bucket.terraform_state.bucket description = "Name of the S3 bucket for Terraform state" } output "state_lock_table_name" { value = aws_dynamodb_table.terraform_locks.name description = "Name of the DynamoDB table for state locking" } ``` **terraform/bootstrap/variables.tf**: ```hcl variable "aws_region" { description = "AWS region for resources" type = string default = "us-east-1" } variable "project_name" { description = "Project name for resource naming" type = string default = "data-infra" } variable "environment" { description = "Environment name" type = string default = "production" } variable "github_org" { description = "GitHub organization or username" type = string } variable "github_repo" { description = "GitHub repository name" type = string } ``` ## Main Infrastructure Configuration **terraform/main/main.tf** (example data infrastructure): ```hcl terraform { required_version = ">= 1.0" backend "s3" { bucket = "data-infra-terraform-state-production" key = "main/terraform.tfstate" region = "us-east-1" dynamodb_table = "data-infra-terraform-locks" encrypt = true } required_providers { aws = { source = "hashicorp/aws" version = "~> 5.0" } } } provider "aws" { region = var.aws_region default_tags { tags = { Project = var.project_name Environment = var.environment ManagedBy = "Terraform" DeployedBy = "GitHub-Actions" } } } # Example: S3 bucket for data lake resource "aws_s3_bucket" "data_lake" { bucket = "${var.project_name}-data-lake-${var.environment}" } resource "aws_s3_bucket_versioning" "data_lake" { bucket = aws_s3_bucket.data_lake.id versioning_configuration { status = "Enabled" } } resource "aws_s3_bucket_server_side_encryption_configuration" "data_lake" { bucket = aws_s3_bucket.data_lake.id rule { apply_server_side_encryption_by_default { sse_algorithm = "AES256" } } } # Example: Glue database for data catalog resource "aws_glue_catalog_database" "analytics" { name = "${var.project_name}_analytics_${var.environment}" description = "Analytics data catalog database" } # Example: IAM role for Glue jobs resource "aws_iam_role" "glue_job" { name = "${var.project_name}-glue-job-role-${var.environment}" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [ { Effect = "Allow" Principal = { Service = "glue.amazonaws.com" } Action = "sts:AssumeRole" } ] }) } resource "aws_iam_role_policy_attachment" "glue_service" { role = aws_iam_role.glue_job.name policy_arn = "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole" } # Outputs output "data_lake_bucket" { value = aws_s3_bucket.data_lake.bucket description = "Name of the data lake S3 bucket" } output "glue_database" { value = aws_glue_catalog_database.analytics.name description = "Name of the Glue catalog database" } ``` ## GitHub Actions Workflows ### CI Workflow: Format and Validation **.github/workflows/ci.yml**: ```yaml name: Terraform CI on: pull_request: branches: - main paths: - 'terraform/**' - '.github/workflows/ci.yml' permissions: contents: read pull-requests: write jobs: terraform-checks: name: Terraform Checks runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v4 - name: Setup Terraform uses: hashicorp/setup-terraform@v3 with: terraform_version: 1.5.0 - name: Terraform Format Check id: fmt run: terraform fmt -check -recursive terraform/ continue-on-error: true - name: Terraform Init (Main) run: terraform -chdir=terraform/main init -backend=false - name: Terraform Validate (Main) run: terraform -chdir=terraform/main validate - name: Comment PR if: steps.fmt.outcome == 'failure' uses: actions/github-script@v7 with: script: | github.rest.issues.createComment({ issue_number: context.issue.number, owner: context.repo.owner, repo: context.repo.repo, body: '❌ Terraform formatting check failed. Run `terraform fmt -recursive terraform/` to fix.' }) - name: Fail if format check failed if: steps.fmt.outcome == 'failure' run: exit 1 ``` ### Deploy Workflow: Plan and Apply **.github/workflows/deploy.yml**: ```yaml name: Deploy Infrastructure on: push: branches: - main paths: - 'terraform/main/**' workflow_dispatch: permissions: id-token: write contents: read pull-requests: write jobs: terraform-plan: name: Terraform Plan runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v4 - name: Configure AWS credentials uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: ${{ secrets.AWS_ROLE_ARN }} aws-region: us-east-1 - name: Setup Terraform uses: hashicorp/setup-terraform@v3 with: terraform_version: 1.5.0 - name: Terraform Init run: terraform -chdir=terraform/main init - name: Terraform Plan id: plan run: | terraform -chdir=terraform/main plan -no-color -out=tfplan terraform -chdir=terraform/main show -no-color tfplan > plan.txt - name: Upload plan uses: actions/upload-artifact@v4 with: name: terraform-plan path: | terraform/main/tfplan plan.txt retention-days: 5 terraform-apply: name: Terraform Apply needs: terraform-plan runs-on: ubuntu-latest environment: production steps: - name: Checkout code uses: actions/checkout@v4 - name: Configure AWS credentials uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: ${{ secrets.AWS_ROLE_ARN }} aws-region: us-east-1 - name: Setup Terraform uses: hashicorp/setup-terraform@v3 with: terraform_version: 1.5.0 - name: Terraform Init run: terraform -chdir=terraform/main init - name: Download plan uses: actions/download-artifact@v4 with: name: terraform-plan path: terraform/main/ - name: Terraform Apply run: terraform -chdir=terraform/main apply -auto-approve tfplan ``` ## Common Workflows ### Adding New Infrastructure 1. **Create/modify Terraform files** in `terraform/main/`: ```hcl # terraform/main/kinesis.tf resource "aws_kinesis_stream" "events" { name = "${var.project_name}-events-${var.environment}" shard_count = 1 retention_period = 24 shard_level_metrics = [ "IncomingBytes", "IncomingRecords", "OutgoingBytes", "OutgoingRecords", ] } output "kinesis_stream_name" { value = aws_kinesis_stream.events.name description = "Name of the Kinesis stream" } ``` 2. **Format Terraform files**: ```bash terraform fmt -recursive terraform/ ``` 3. **Validate locally** (optional but recommended): ```bash terraform -chdir=terraform/main init -backend=false terraform -chdir=terraform/main validate