cloud-platforms

name	cloud-platforms
description	AWS, GCP, Azure data platforms, infrastructure as code, and cloud-native data solutions
sasmp_version	1.3.0
bonded_agent	05-cloud-engineer
bond_type	PRIMARY_BOND
skill_version	2.0.0
last_updated	2025-01
complexity	intermediate
estimated_mastery_hours	160
prerequisites	["python-programming","sql-databases"]
unlocks	["containerization","big-data","data-warehousing"]

Cloud Platforms for Data Engineering

Production-grade cloud infrastructure for data pipelines, storage, and analytics on AWS, GCP, and Azure.

Quick Start

# AWS S3 + Lambda Data Pipeline
import boto3
import json

s3_client = boto3.client('s3')
glue_client = boto3.client('glue')

def lambda_handler(event, context):
    """Process S3 event and trigger Glue job."""
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']

    # Trigger Glue ETL job
    response = glue_client.start_job_run(
        JobName='etl-process-raw-data',
        Arguments={
            '--source_path': f's3://{bucket}/{key}',
            '--output_path': 's3://processed-bucket/output/'
        }
    )

    return {'statusCode': 200, 'jobRunId': response['JobRunId']}

Core Concepts

1. AWS Data Stack

# AWS Glue ETL Job (PySpark)
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
 awsglue.job  Job

args = getResolvedOptions(sys.argv, [, , ])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args[], args)


df = glueContext.create_dynamic_frame.from_options(
    connection_type=,
    connection_options={: [args[]]},
    =
)


df_transformed = ApplyMapping.apply(
    frame=df,
    mappings=[
        (, , , ),
        (, , , ),
        (, , , )
    ]
)


glueContext.write_dynamic_frame.from_options(
    frame=df_transformed,
    connection_type=,
    connection_options={
        : args[],
        : [, ]
    },
    =
)

job.commit()