message-queues

# Message Queues Design reliable message-driven systems. ## Queue Types | Queue | Persistence | Ordering | Use Case | |-------|------------|----------|----------| | RabbitMQ | Optional | Per queue | Task distribution, RPC | | Apache Kafka | Durable (disk) | Per partition | Event streaming, logs | | AWS SQS | Durable | Best effort (std) / Strict (FIFO) | Serverless decoupling | | Redis Pub/Sub | None | Per channel | Real-time notifications | ## RabbitMQ Patterns ### Work Queues (Competing Consumers) ``` Producer → Queue → Consumer 1 → Consumer 2 → Consumer 3 ``` - Messages distributed round-robin - Ack on success, nack on failure (requeue or DLQ) - Prefetch count controls concurrency ### Pub/Sub (Exchange → Binding → Queue) - Fanout: broadcast to all queues - Direct: route by routing key - Topic: route by pattern (user.*, user.created) - Headers: route by header values ## Kafka Patterns ### Topics & Partitions - Messages within a partition are ordered - Partitions enable parallelism - Consumer group = one instance per partition ### Producer ```javascript await producer.send({ topic: 'order-events', messages: [{ key: orderId, value: JSON.stringify(order) }], }); ``` ### Consumer ```javascript await consumer.run({ eachMessage: async ({ topic, partition, message }) => { await processOrder(message.value); }, }); ``` ## Dead Letter Queues - Messages that can't be processed go to DLQ - Analyze DLQ periodically for systemic issues - DLQ messages can be replayed after fix - Set max retry count before DLQ ## Best Practices - Idempotent consumers (same message processed twice = safe) - Monitor queue depth, consumer lag, error rate - Set message TTL to prevent infinite backlog - Use structured message schemas (Avro, Protobuf) - Test with network failures and consumer crashes