// Guides use of AWS messaging and streaming services. Covers Amazon SQS, Amazon SNS, Amazon EventBridge, Amazon MQ, Amazon Kinesis Data Streams, Amazon Data Firehose, Amazon Managed Service for Apache Flink, and Amazon Managed Streaming for Apache Kafka (MSK). Use when implementing messaging and streaming patterns.
Builds generative AI applications on Amazon Bedrock. Covers model invocation (Converse API, InvokeModel), RAG with Knowledge Bases, Bedrock Agents, Guardrails, and AgentCore. Use when invoking models, setting up Knowledge Bases, creating agents, applying guardrails, deploying to AgentCore, troubleshooting Bedrock errors (ThrottlingException, AccessDeniedException), or choosing models (Claude, Llama, Nova, Titan). ALSO USE for prompt caching setup and debugging, quota health checks and throttling diagnosis, cost attribution and tracking, migrating between Claude model generations (4.5 to 4.6 to 4.7), chunking strategies, API selection (Converse vs InvokeModel), guardrail capabilities, and model selection. NOT for custom model training, Rekognition, or Comprehend.
Analyze AWS costs, find savings, manage budgets, evaluate Savings Plans and Reserved Instances, right-size EC2/Lambda/RDS/EBS with Compute Optimizer, look up service pricing, query CUR with Athena, detect cost anomalies, scope costs to billing views, and monitor Free Tier usage. Triggers on: AWS bill, cost analysis, reduce spend, savings plan, reserved instance, right-size, budget alert, cost optimization, pricing, free tier, cost anomaly, CUR, cost audit, billing view, billing view ARN.
Authors, deploys, and troubleshoots AWS infrastructure using CDK with TypeScript or Python. Covers best practices, stack architecture, and construct patterns. Always use when writing CDK constructs, bootstrapping environments, running cdk deploy/synth/diff, fixing CDK or CloudFormation errors, planning stack structure, importing existing resources, resolving drift, or refactoring stacks without resource replacement.
Author, validate, and troubleshoot AWS CloudFormation templates. Covers template authoring with secure defaults, pre-deployment validation (cfn-lint, cfn-guard, change sets), and root-cause diagnosis of failed stacks using CloudFormation events and CloudTrail correlation.
Deploys and operates containerized workloads on ECS, Fargate, and ECR. Covers task definitions, Fargate services, ECR repository setup and lifecycle policies, ECS Exec debugging, service scaling, deployment strategies, load balancer integration, and logging configuration. Use when deploying, debugging, or optimizing containers on AWS. ALSO USE for container deployment options (ECS vs ECS Express Mode), networking modes, health check troubleshooting, OOM errors, secrets injection, blue/green deployments, ECR image management, and App Runner sunset guidance and migration. NOT for Kubernetes, EKS, or CI/CD pipelines.
Verified corrections for IAM behaviors that AI agents frequently get wrong — policy evaluation edge cases, trust policy gotchas, STS session limits, Organizations quirks, and SAML/MFA specifics. Use alongside documentation when working with IAM roles, policies, STS, or Organizations. Do NOT use for non-IAM authorization like Cognito user-pool policies or app-level RBAC.
| name | aws-messaging-and-streaming |
| description | Guides use of AWS messaging and streaming services. Covers Amazon SQS, Amazon SNS, Amazon EventBridge, Amazon MQ, Amazon Kinesis Data Streams, Amazon Data Firehose, Amazon Managed Service for Apache Flink, and Amazon Managed Streaming for Apache Kafka (MSK). Use when implementing messaging and streaming patterns. |
| version | 1 |
When answering AWS messaging and streaming questions, verify specific numbers, versions, limits, and behavioral details from service-specific skills or official AWS documentation. When uncertain, search skills or docs rather than guessing. Fabricated configuration options or incorrect version numbers are worse than admitting uncertainty.
When a question asks about recommended configurations (CloudWatch alarm settings, thresholds, missing data treatment), search for the service-specific skills or documentation rather than relying on general best practices.
Domain expertise for choosing and using AWS services that move data between producers and consumers. This skill covers two fundamental patterns — messaging and streaming — and the AWS services that implement each. Use this skill to decide which pattern fits a workload, select the right service, and understand how services integrate with each other.
For specific guidance on individual AWS services, see reference files or service-specific Skills.
Messaging enables decoupled, asynchronous communication between components. A producer sends a message; one or more consumers receive and process it. Once processed, the message is typically deleted. Messaging services handle delivery guarantees, retries, and dead-letter routing.
Key characteristics:
Streaming enables ordered, durable, high-throughput continuous data flow. Producers append records to a log; consumers read from positions in that log. Records persist for a configurable retention period regardless of consumption.
Key characteristics:
| Dimension | Messaging | Streaming |
|---|---|---|
| Data lifecycle | Deleted after consumption | Retained for replay (hours to indefinitely) |
| Ordering | Best-effort (Standard) or per-group (FIFO) | Strict per-partition/shard |
| Consumer model | Competing consumers (work distribution) | Independent readers (fan-out by position) |
| Throughput pattern | Bursty, variable | Sustained, high-volume |
| Replay | Not supported (except DLQ redrive) | Native — seek to any position in retention |
| Typical latency | Milliseconds (push or short-poll) | Milliseconds to low seconds |
| Scaling unit | Concurrency (consumers/pollers) | Partitions or shards |
These services are generally used for messaging workloads. Sometimes streaming services (Kinesis Data Streams, Managed Streaming for Apache Kafka) are also used for messaging workloads, depending on exact use case and requirements.
| Service | Best For | Key Differentiator |
|---|---|---|
| Amazon SQS | Task queues, decoupling, buffering | Fully managed, unlimited throughput (Standard), exactly-once (FIFO), fair queues for multi-tenant workloads |
| Amazon SNS | Fan-out, pub/sub notifications | Push to multiple subscribers (SQS, Lambda, HTTP, email, SMS) |
| Amazon EventBridge | Event routing, cross-account/SaaS integration | Content-based filtering, schema registry, 200+ AWS source integrations |
| Amazon MQ | Lift-and-shift of existing JMS/AMQP/MQTT apps | Protocol compatibility (ActiveMQ, RabbitMQ) for legacy migration |
These services are generally used for streaming workloads.
| Service | Best For | Key Differentiator |
|---|---|---|
| Amazon Kinesis Data Streams | Real-time ingestion with AWS-native consumers | On-demand Advantage mode (instant scaling, no shard management), 1–365 day retention |
| Amazon Data Firehose | Zero-admin delivery to storage/analytics | Auto-scales, buffers, batches, and delivers to destinations |
| Amazon Managed Service for Apache Flink | Complex stream processing (joins, windows, state) | Full Apache Flink runtime — SQL, Java, Python APIs for stateful computation |
| Amazon MSK | Kafka-native workloads, ecosystem compatibility | Apache Kafka API, Express brokers (3x throughput, 20x faster scaling compared to Standard brokers), broad connector ecosystem |
SQS system vs. user message attributes: Attributes like AWSTraceHeader (set by X-Ray / EventBridge / Pipes when sending to an SQS DLQ) and SenderId, SentTimestamp are SQS system attributes, NOT user message attributes. They are never returned by default from ReceiveMessage — request them explicitly via AttributeNames=[...] (or MessageSystemAttributeNames), separate from MessageAttributeNames which fetches user attributes. This matters for DLQs, where the trace header rides on the system attribute and the user-attributes slot carries the service's failure metadata (e.g. EventBridge's RULE_ARN, ERROR_CODE).
SNS → Firehose → S3 record separator: For SNS subscriptions using the firehose protocol that land in S3, records are already newline-delimited by default (NDJSON). Do NOT turn on Firehose's AppendDelimiterToRecord — SNS emits the newline itself, and enabling the processor produces double newlines.
EventBridge rule target DLQ + SNS subscription DLQ both need a DLQ queue policy. Attaching the DLQ alone is not enough — the DLQ silently drops messages until its queue policy allows the service principal. EventBridge: PutTargets with DeadLetterConfig.Arn=<DLQ>, plus SQS policy Allow sqs:SendMessage for Service: events.amazonaws.com with aws:SourceArn = the rule ARN. SNS: SetSubscriptionAttributes RedrivePolicy={"deadLetterTargetArn":"<DLQ>"}, plus SQS policy allowing Service: sns.amazonaws.com scoped by the topic ARN.
SQS production defaults: long polling + customer-managed encryption. New queues default to short-poll (ReceiveMessageWaitTimeSeconds=0) and SSE-SQS (AWS-owned key). For production, SetQueueAttributes with ReceiveMessageWaitTimeSeconds=20 (long polling) and KmsMasterKeyId=<customer-managed key id/ARN> rather than leaving alias/aws/sqs.
Broker and Kafka credentials belong in Secrets Manager, not connection strings. Do not hardcode usernames, passwords, or SASL/SCRAM credentials in application config, env vars, JAAS files, or IaC. For Amazon MQ (ActiveMQ/RabbitMQ) store broker users as secrets and fetch at startup; Lambda event source mappings for Amazon MQ require the broker credentials to be supplied as a Secrets Manager secret ARN (BASIC_AUTH), not inline. For MSK SASL/SCRAM the secret is not optional: it must be named with the AmazonMSK_ prefix and encrypted with a customer-managed KMS key (secrets created with the default aws/secretsmanager key cannot be associated with a cluster), then attached via BatchAssociateScramSecret. Lambda event source mappings for MSK (SASL/SCRAM or mTLS) and self-managed Kafka also reference a Secrets Manager secret ARN rather than inline credentials. Enable rotation and scope IAM read access (secretsmanager:GetSecretValue) to the consuming role only. See AWS Well-Architected SEC02-BP03 Store and use secrets securely.
Service-principal resource policies need aws:SourceArn / aws:SourceAccount conditions. When a queue or topic policy grants a service principal like events.amazonaws.com, sns.amazonaws.com, or s3.amazonaws.com permission to sqs:SendMessage or sns:Publish, omitting source conditions opens a confused-deputy hole — any rule, topic, or bucket in any AWS account can drive writes. Scope every such statement with aws:SourceArn (the specific rule/topic/bucket/pipe ARN; use ArnLike with * when the ARN isn't fully known yet) and aws:SourceAccount (your account ID). For S3 event notifications both keys are required because S3 bucket ARNs don't carry the account ID, so aws:SourceArn alone doesn't constrain the account. The same pattern applies to role trust policies for IAM roles used by EventBridge rules and EventBridge Pipes (principal events.amazonaws.com / pipes.amazonaws.com, aws:SourceArn = the rule or pipe ARN) — not just the DLQ case called out above. See the IAM User Guide on The confused deputy problem.
