message-queues

name	message-queues
description	RabbitMQ, Kafka, SQS, pub/sub, competing consumers, dead letter queues, and event streaming
metadata	{"author":"cosmicstack-labs","version":"1.0.0","category":"backend","tags":["message-queues","kafka","rabbitmq","sqs","event-streaming"]}

Message Queues

Design reliable message-driven systems.

Queue Types

Queue	Persistence	Ordering	Use Case
RabbitMQ	Optional	Per queue	Task distribution, RPC
Apache Kafka	Durable (disk)	Per partition	Event streaming, logs
AWS SQS	Durable	Best effort (std) / Strict (FIFO)	Serverless decoupling
Redis Pub/Sub	None	Per channel	Real-time notifications

RabbitMQ Patterns

Work Queues (Competing Consumers)

Producer → Queue → Consumer 1
                 → Consumer 2
                 → Consumer 3

• Messages distributed round-robin
• Ack on success, nack on failure (requeue or DLQ)
• Prefetch count controls concurrency

Pub/Sub (Exchange → Binding → Queue)

• Fanout: broadcast to all queues
• Direct: route by routing key
• Topic: route by pattern (user.*, user.created)
• Headers: route by header values

Kafka Patterns

Topics & Partitions

• Messages within a partition are ordered
• Partitions enable parallelism
• Consumer group = one instance per partition

Producer

await producer.send({
  topic: 'order-events',
  messages: [{ key: orderId, value: JSON.stringify(order) }],
});

Consumer

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    await processOrder(message.value);
  },
});

Dead Letter Queues

• Messages that can't be processed go to DLQ
• Analyze DLQ periodically for systemic issues
• DLQ messages can be replayed after fix
• Set max retry count before DLQ

Best Practices

• Idempotent consumers (same message processed twice = safe)
• Monitor queue depth, consumer lag, error rate
• Set message TTL to prevent infinite backlog
• Use structured message schemas (Avro, Protobuf)
• Test with network failures and consumer crashes