design-serverless-observability

name	design-serverless-observability
description	Use when operating serverless functions in production — implementing structured logging, distributed tracing, anomaly-based alerting, and cold start monitoring to detect security incidents and performance degradation.
source	OWASP Serverless Top 10 SLS-5 (owasp.org/www-project-serverless-top-10/); AWS Lambda Powertools documentation; AWS Well-Architected Operational Excellence Pillar; Datadog State of Serverless 2023
tags	["security","owasp","serverless","lambda","observability","logging","tracing","cloud"]

Design Serverless Observability

Implement structured JSON logs with correlation IDs, distributed tracing with AWS X-Ray, anomaly-based CloudWatch alarms, and security-relevant event logging — enabling incident detection and forensic investigation in serverless architectures where traditional server logs don't exist.

Why This Is Best Practice

Adopted by: OWASP Serverless Top 10 SLS-5 (Inadequate Function Monitoring and Logging). AWS Lambda Powertools (used by thousands of teams for production Lambda observability) is the reference implementation. AWS Well-Architected Operational Excellence Pillar mandates distributed tracing for all serverless workloads. Datadog's "State of Serverless" (2023) found that teams using structured logging and distributed tracing resolve incidents 3× faster than those using unstructured logs. Impact: Unlike EC2/ECS, serverless functions produce logs only in CloudWatch — there's no server to SSH into, no persistent process to attach a debugger to, and no network-level visibility without explicit instrumentation. The 2022 Lacework serverless threat research found that 60% of Lambda security incidents were detected only through billing anomalies (sudden cost spikes from unexpected invocations) rather than from application logs — because most teams lack structured Lambda security logging. Without correlation IDs, a multi-function execution chain is nearly impossible to trace after an incident. Why best: Unstructured print/console logs are unsearchable at scale and can't be correlated across functions. Structured JSON logs with consistent fields (request_id, user_id, function_name, duration) enable CloudWatch Insights queries, automated anomaly detection, and SIEM ingestion. Distributed tracing with X-Ray links a user request through API Gateway → Lambda A → DynamoDB → Lambda B — critical for security forensics in event-driven architectures.

Sources: OWASP Serverless Top 10 SLS-5; AWS Lambda Powertools documentation; AWS Well-Architected Operational Excellence Pillar; Datadog State of Serverless (2023)

Steps

1. Use AWS Lambda Powertools for structured logging:

   from aws_lambda_powertools import Logger, Tracer, Metrics
   from aws_lambda_powertools.metrics import MetricUnit
   from aws_lambda_powertools.utilities.typing import LambdaContext
   
   logger = Logger(service="order-processor")
   tracer = Tracer(service="order-processor")
   metrics = Metrics(namespace="OrderService", service="order-processor")
   
   @logger.inject_lambda_context(correlation_id_path="requestContext.requestId", log_event=False)
   @tracer.capture_lambda_handler
   @metrics.log_metrics(capture_cold_start_metric=True)
   def lambda_handler(event: dict, context: LambdaContext) -> dict:
       order_id = event["pathParameters"]["orderId"]
   
       # Structured log with consistent fields
       logger.info("Processing order", extra={"order_id": order_id, "user_id": event["requestContext"]["authorizer"]["userId"]})
   
       try:
           result = process_order(order_id)
           metrics.add_metric(name="OrdersProcessed", unit=MetricUnit.Count, value=1)
           return {"statusCode": 200, "body": json.dumps(result)}
       except OrderNotFoundError:
           logger.warning("Order not found", extra={"order_id": order_id})
            {: , : }
   

Rules

• Always inject correlation_id into logs — without it, tracing a request across multiple functions in a chain requires guessing from timestamps.
• Never log event payloads at INFO level — use DEBUG level for payloads and ensure DEBUG is disabled in production (LOG_LEVEL: INFO).
• Security events (authentication, authorization decisions, data access) must be logged even when no error occurs — security logging is separate from error logging.
• CloudWatch Log Groups must have retention policies — without them, logs accumulate indefinitely and are never purged (cost and compliance risk).

Common Mistakes

• Using print() instead of structured logger — print output is a single string in CloudWatch; can't be queried by field, can't be shipped to SIEM without parsing.
• Not sampling X-Ray in high-traffic functions — X-Ray at 100% sampling adds latency and cost at scale; use reservoir + fixed rate sampling in production.
• Separate alarms per function instead of service-level dashboards — Lambda functions are often orchestrated; alert on service-level error rates, not individual functions.
• No log retention policy on CloudWatch Log Groups — Lambda automatically creates log groups with no expiry; set retention to 90 days to balance cost and forensic investigation window.