| name | message-queues |
| description | RabbitMQ, Kafka, SQS, pub/sub, competing consumers, dead letter queues, and event streaming |
| metadata | {"author":"cosmicstack-labs","version":"1.0.0","category":"backend","tags":["message-queues","kafka","rabbitmq","sqs","event-streaming"]} |
Message Queues
Design reliable message-driven systems.
Queue Types
| Queue | Persistence | Ordering | Use Case |
|---|
| RabbitMQ | Optional | Per queue | Task distribution, RPC |
| Apache Kafka | Durable (disk) | Per partition | Event streaming, logs |
| AWS SQS | Durable | Best effort (std) / Strict (FIFO) | Serverless decoupling |
| Redis Pub/Sub | None | Per channel | Real-time notifications |
RabbitMQ Patterns
Work Queues (Competing Consumers)
Producer → Queue → Consumer 1
→ Consumer 2
→ Consumer 3
- Messages distributed round-robin
- Ack on success, nack on failure (requeue or DLQ)
- Prefetch count controls concurrency
Pub/Sub (Exchange → Binding → Queue)
- Fanout: broadcast to all queues
- Direct: route by routing key
- Topic: route by pattern (user.*, user.created)
- Headers: route by header values
Kafka Patterns
Topics & Partitions
- Messages within a partition are ordered
- Partitions enable parallelism
- Consumer group = one instance per partition
Producer
await producer.send({
topic: 'order-events',
messages: [{ key: orderId, value: JSON.stringify(order) }],
});
Consumer
await consumer.run({
eachMessage: async ({ topic, partition, message }) => {
await processOrder(message.value);
},
});
Dead Letter Queues
- Messages that can't be processed go to DLQ
- Analyze DLQ periodically for systemic issues
- DLQ messages can be replayed after fix
- Set max retry count before DLQ
Best Practices
- Idempotent consumers (same message processed twice = safe)
- Monitor queue depth, consumer lag, error rate
- Set message TTL to prevent infinite backlog
- Use structured message schemas (Avro, Protobuf)
- Test with network failures and consumer crashes
