aws-sqs

# AWS SQS Amazon Simple Queue Service (SQS) is a fully managed message queuing service for decoupling and scaling microservices, distributed systems, and serverless applications. ## Table of Contents - [Core Concepts](#core-concepts) - [Common Patterns](#common-patterns) - [CLI Reference](#cli-reference) - [Best Practices](#best-practices) - [Troubleshooting](#troubleshooting) - [References](#references) ## Core Concepts ### Queue Types | Type | Description | Use Case | |------|-------------|----------| | **Standard** | At-least-once, best-effort ordering | High throughput | | **FIFO** | Exactly-once, strict ordering | Order-sensitive processing | ### Key Settings | Setting | Description | Default | |---------|-------------|---------| | **Visibility Timeout** | Time message is hidden after receive | 30 seconds | | **Message Retention** | How long messages are kept | 4 days (max 14) | | **Delay Seconds** | Delay before message is available | 0 | | **Max Message Size** | Maximum message size | 256 KB | ### Dead-Letter Queue (DLQ) Queue for messages that failed processing after maxReceiveCount attempts. ## Common Patterns ### Create a Standard Queue **AWS CLI:** ```bash aws sqs create-queue \ --queue-name my-queue \ --attributes '{ "VisibilityTimeout": "60", "MessageRetentionPeriod": "604800", "ReceiveMessageWaitTimeSeconds": "20" }' ``` **boto3:** ```python import boto3 sqs = boto3.client('sqs') response = sqs.create_queue( QueueName='my-queue', Attributes={ 'VisibilityTimeout': '60', 'MessageRetentionPeriod': '604800', 'ReceiveMessageWaitTimeSeconds': '20' # Long polling } ) queue_url = response['QueueUrl'] ``` ### Create FIFO Queue ```bash aws sqs create-queue \ --queue-name my-queue.fifo \ --attributes '{ "FifoQueue": "true", "ContentBasedDeduplication": "true" }' ``` ### Configure Dead-Letter Queue ```bash # Create DLQ aws sqs create-queue --queue-name my-queue-dlq # Get DLQ ARN DLQ_ARN=$(aws sqs get-queue-attributes \ --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue-dlq \ --attribute-names QueueArn \ --query 'Attributes.QueueArn' --output text) # Set redrive policy on main queue aws sqs set-queue-attributes \ --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue \ --attributes "{ \"RedrivePolicy\": \"{\\\"deadLetterTargetArn\\\":\\\"${DLQ_ARN}\\\",\\\"maxReceiveCount\\\":\\\"3\\\"}\" }" ``` ### Send Messages ```python import boto3 import json sqs = boto3.client('sqs') queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue' # Send single message sqs.send_message( QueueUrl=queue_url, MessageBody=json.dumps({'order_id': '12345', 'action': 'process'}), MessageAttributes={ 'MessageType': { 'DataType': 'String', 'StringValue': 'Order' } } ) # Send to FIFO queue sqs.send_message( QueueUrl='https://sqs.us-east-1.amazonaws.com/123456789012/my-queue.fifo', MessageBody=json.dumps({'order_id': '12345'}), MessageGroupId='order-12345', MessageDeduplicationId='unique-id-12345' ) # Batch send (up to 10 messages) sqs.send_message_batch( QueueUrl=queue_url, Entries=[ {'Id': '1', 'MessageBody': json.dumps({'id': 1})}, {'Id': '2', 'MessageBody': json.dumps({'id': 2})}, {'Id': '3', 'MessageBody': json.dumps({'id': 3})} ] ) ``` ### Receive and Process Messages ```python import boto3 import json sqs = boto3.client('sqs') queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue' while True: # Long polling (wait up to 20 seconds) response = sqs.receive_message( QueueUrl=queue_url, MaxNumberOfMessages=10, WaitTimeSeconds=20, MessageAttributeNames=['All'], AttributeNames=['All'] ) messages = response.get('Messages', []) for message in messages: try: body = json.loads(message['Body']) print(f"Processing: {body}") # Process message... # Delete on success sqs.delete_message( QueueUrl=queue_url, ReceiptHandle=message['ReceiptHandle'] ) except Exception as e: print(f"Error processing message: {e}") # Message will become visible again after visibility timeout ``` ### Lambda Integration ```bash # Create event source mapping aws lambda create-event-source-mapping \ --function-name my-function \ --event-source-arn arn:aws:sqs:us-east-1:123456789012:my-queue \ --batch-size 10 \ --maximum-batching-window-in-seconds 5 ``` Lambda handler: ```python import json def handler(event, context): for record in event['Records']: body = json.loads(record['body']) message_id = record['messageId'] try: process_message(body) except Exception as e: raise # Re-raise to put message back in queue return {'batchItemFailures': []} ``` ## CLI Reference ### Queue Management | Command | Description | |---------|-------------| | `aws sqs create-queue` | Create queue | | `aws sqs delete-queue` | Delete queue | | `aws sqs list-queues` | List queues | | `aws sqs get-queue-url` | Get queue URL by name | | `aws sqs get-queue-attributes` | Get queue settings | | `aws sqs set-queue-attributes` | Update queue settings | ### Messaging | Command | Description | |---------|-------------| | `aws sqs send-message` | Send single message | | `aws sqs send-message-batch` | Send up to 10 messages | | `aws sqs receive-message` | Receive messages | | `aws sqs delete-message` | Delete message | | `aws sqs delete-message-batch` | Delete up to 10 messages | | `aws sqs purge-queue` | Delete all messages | ### Visibility | Command | Description | |---------|-------------| | `aws sqs change-message-visibility` | Change timeout | | `aws sqs change-message-visibility-batch` | Batch change | ## Best Practices ### Message Processing - **Use long polling** (WaitTimeSeconds=20) to reduce API calls - **Delete messages promptly** after successful processing - **Configure appropriate visibility timeout** (> processing time) - **Implement idempotent consumers** for at-least-once delivery ### Dead-Letter Queues - **Always configure DLQ** for production queues - **Set appropriate maxReceiveCount** (usually 3-5) - **Monitor DLQ depth** with CloudWatch alarms - **Process DLQ messages** manually or with automation ### FIFO Queues - **Use message group IDs** to partition ordering - **Enable content-based deduplication** or provide dedup IDs - **Throughput**: 300 msgs/sec without batching, 3000 with ### Security - **Use queue policies** to control access - **Enable encryption** with SSE-SQS or SSE-KMS - **Use VPC endpoints** for private access ## Troubleshooting ### Messages Not Being Received **Debug:** ```bash # Check approximate message counts aws sqs get-queue-attributes \ --queue-url $QUEUE_URL \ --attribute-names \ ApproximateNumberOfMessages,\ ApproximateNumberOfMessagesNotVisible,\ ApproximateNumberOfMessagesDelayed ``` ### Messages Going to DLQ **Redrive from DLQ:** ```bash # Start redrive aws sqs start-message-move-task \ --source-arn arn:aws:sqs:us-east-1:123456789012:my-queue-dlq \ --destination-arn arn:aws:sqs:us-east-1:123456789012:my-queue ``` ### Duplicate Processing **Solutions:** - Use FIFO queues for exactly-once - Implement idempotency in consumer - Track processed message IDs in database ### Lambda Not Processing ```bash # Check event source mapping aws lambda list-event-source-mappings \ --function-name my-function # Check for errors aws lambda get-event-source-mapping \ --uuid <mapping-uuid> ``` ## References - [SQS Developer Guide](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/) - [SQS API Reference](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/) - [SQS CLI Reference](https://docs.aws.amazon.com/cli/latest/reference/sqs/) - [boto3 SQS](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs.html)