aws-logging-diagnostics

name	aws-logging-diagnostics
description	Querying and analyzing CloudWatch Logs, CloudTrail events, and other AWS log sources. Use when investigating errors, auditing actions, or understanding what happened with an AWS resource.

AWS Logging & Diagnostics Skill

Security: Always ensure migrated resources meet or exceed the security configuration of the source resources. Refer to SECURITY.md for security requirements.

CloudWatch Logs

Find Log Groups

# List log groups matching a pattern
aws logs describe-log-groups --log-group-name-prefix "/aws/lambda/<func>" --region <region> --query 'logGroups[*].{Name:logGroupName,Stored:storedBytes,Retention:retentionInDays}'

# List recent log streams
aws logs describe-log-streams --log-group-name <group> --order-by LastEventTime --descending --limit 5 --region <region> --query 'logStreams[*].{Name:logStreamName,LastEvent:lastEventTimestamp}'

Query Logs (Insights)

# Search for errors in last hour
aws logs start-query --log-group-name <group> --start-time $(date -d '1 hour ago' +%s) --end-time $(date +%s) --query-string 'fields @timestamp, @message | filter @message like /ERROR|Exception|FATAL/ | sort @timestamp desc | limit 50' --region <region>

# Get query results
aws logs get-query-results --query-id <query-id> --region <region>

# Search for specific request/correlation ID
aws logs start-query --log-group-name <group> --start-time $(date -d '24 hours ago' +%s) --end-time $(date +%s) --query-string 'fields @timestamp, @message | filter @message like /<request-id>/ | sort @timestamp desc' --region <region>

# Aggregate error counts
aws logs start-query --log-group-name <group> --start-time $(date -d '24 hours ago' +%s) --end-time $(date +%s) --query-string 'fields @message | filter @message like /ERROR/ | stats count() by bin(1h)' --region <region>

Tail Logs (Live)

aws logs tail <group> --follow --since 5m --region <region>

# With filter
aws logs tail <group> --follow --since 10m --filter-pattern "ERROR" --region <region>

Get Specific Log Events

aws logs get-log-events --log-group-name <group> --log-stream-name <stream> --start-time $(date -d '1 hour ago' +%s000) --region <region> --query 'events[*].{Time:timestamp,Msg:message}'

Filter Patterns

# Filter for JSON fields
aws logs filter-log-events --log-group-name <group> --filter-pattern '{$.statusCode = 500}' --start-time $(date -d '1 hour ago' +%s000) --region <region>

# Filter for text patterns
aws logs filter-log-events --log-group-name <group> --filter-pattern '"OutOfMemory"' --start-time $(date -d '6 hours ago' +%s000) --region <region>

AWS CloudTrail

CloudTrail Security Responsibilities

Customers are responsible for enabling AWS CloudTrail, configuring trail settings, and monitoring audit logs. AWS is responsible for the CloudTrail service infrastructure and API event delivery.

Lookup Recent Events

# Who did what in the last hour
aws cloudtrail lookup-events --start-time $(date -d '1 hour ago' -Iseconds) --region <region> --query 'Events[*].{Time:EventTime,Name:EventName,User:Username,Source:EventSource}'

# Filter by event name
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=<event> --start-time $(date -d '24 hours ago' -Iseconds) --region <region>

# Filter by resource
aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=<resource-name> --start-time $(date -d '7 days ago' -Iseconds) --region <region>

# Filter by user
aws cloudtrail lookup-events --lookup-attributes AttributeKey=Username,AttributeValue=<username> --start-time $(date -d '24 hours ago' -Iseconds) --region <region>

Query CloudTrail via Athena (for historical analysis)

# If CloudTrail logs are in S3, query with Athena
aws athena start-query-execution --query-string "
  SELECT eventtime, eventname, useridentity.arn, sourceipaddress, errorcode, errormessage
  FROM cloudtrail_logs
  WHERE eventname = '<event>'
    AND eventtime > '<date>'
  ORDER BY eventtime DESC
  LIMIT 100
" --query-execution-context Database=<db> --result-configuration OutputLocation=s3://<bucket>/athena-results/ --region <region>

CloudTrail Insights

# Check if insights are enabled
aws cloudtrail get-insight-selectors --trail-name <trail> --region <region>

# Lookup insight events (unusual API activity)
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=<service>.amazonaws.com --start-time $(date -d '7 days ago' -Iseconds) --region <region>

CloudWatch Metrics (Quick Diagnostics)

# EC2 CPU utilization
aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name CPUUtilization --dimensions Name=InstanceId,Value=<id> --start-time $(date -d '1 hour ago' -Iseconds) --end-time $(date -Iseconds) --period 300 --statistics Average --region <region>

# RDS connections
aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name DatabaseConnections --dimensions Name=DBInstanceIdentifier,Value=<id> --start-time $(date -d '1 hour ago' -Iseconds) --end-time $(date -Iseconds) --period 300 --statistics Maximum --region <region>

# Lambda errors
aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Errors --dimensions Name=FunctionName,Value=<func> --start-time $(date -d '1 hour ago' -Iseconds) --end-time $(date -Iseconds) --period 300 --statistics Sum --region <region>

# ALB 5xx errors
aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB --metric-name HTTPCode_ELB_5XX_Count --dimensions Name=LoadBalancer,Value=<alb-id> --start-time $(date -d '1 hour ago' -Iseconds) --end-time $(date -Iseconds) --period 300 --statistics Sum --region <region>

CloudWatch Alarms Check

# List alarms in ALARM state
aws cloudwatch describe-alarms --state-value ALARM --region <region> --query 'MetricAlarms[*].{Name:AlarmName,Metric:MetricName,State:StateValue,Reason:StateReason}'

# Check specific alarm
aws cloudwatch describe-alarms --alarm-names <alarm-name> --region <region>

# Alarm history
aws cloudwatch describe-alarm-history --alarm-name <alarm-name> --history-item-type StateUpdate --region <region>

Diagnostic Pattern

When investigating an issue:

1. What happened: CloudTrail — who/what triggered the action
2. What it said: CloudWatch Logs — error messages, stack traces
3. What it looked like: CloudWatch Metrics — CPU, memory, connections, errors
4. What's alerting: CloudWatch Alarms — any alarms in ALARM state
5. Correlate: match timestamps across all sources

Data Access Logging

Customers are responsible for enabling access logging for Amazon S3 buckets, configuring database audit logging, encrypting Amazon CloudWatch Logs with AWS KMS, and managing log retention policies. AWS is responsible for the underlying logging service infrastructure and log delivery mechanisms.

Enable comprehensive access logging for data operations:

• S3: Enable S3 server access logging or CloudTrail data events for buckets containing sensitive data
• RDS: Enable database audit logging (e.g., pgaudit for PostgreSQL, audit plugin for MySQL)
• EBS: CloudTrail logs EBS snapshot and volume operations by default
• CloudWatch Logs: Use AWS Key Management Service (AWS KMS) encryption for log groups containing sensitive data: aws logs associate-kms-key --log-group-name <group> --kms-key-id <key-arn>