realtime-scaling

name	realtime-scaling
description	Scale a realtime backend horizontally with pub/sub fanout and presence tracking. Use when moving past a single-node WebSocket/SSE server.

Realtime Scaling

Instructions

Realtime backends are stateful: each user's connection lives on one node. Horizontal scale is a fanout + routing problem, not a stateless load-balance problem.

1. Node Anatomy

Each realtime node holds:

• A pool of open connections (WebSocket or SSE).
• A map from channel -> [connection] (local subscribers).
• A reverse map from connection -> [channel] (for cleanup on disconnect).

No user state lives only in one node -- persist whatever must survive a node restart.

2. Pub/Sub Fanout

Every business-side event publishes once to a pub/sub bus; every node filters against its local subscriber map.

publisher -> bus(channel=X) -> [nodeA, nodeB, nodeC]
                                    |        |        |
                                    v        v        v
                           matching local connections dispatch

Options:

• Redis Pub/Sub: simplest, at-most-once, best for small-to-mid scale.
• Redis Streams: durable, consumer groups, good for replay.
• NATS: low-latency, subject wildcards, clustered.
• Kafka: durable, ordered per partition, best for analytics-grade fanout.

For most mobile apps under a few hundred thousand concurrent connections, Redis (Pub/Sub + Streams) is sufficient.

3. Channel Naming

Hierarchical, coarse first:

user:{user_id}             -- direct messages, badges
thread:{thread_id}         -- chat channel
entity:{type}:{id}         -- per-object updates
broadcast:{tenant}         -- fanout to all users in a tenant

Avoid per-connection channels -- they do not fan out and increase subscription churn.

4. Subscription Management

When a client subscribes, the node checks whether it is already subscribed to that bus topic. Maintain a refcount:

func (n *Node) Subscribe(conn *Conn, ch string) {
    if n.refs[ch] == 0 { n.bus.Subscribe(ch) }
    n.refs[ch]++
    n.channels[ch] = append(n.channels[ch], conn)
}

func (n *Node) Unsubscribe(conn *Conn, ch string) {
    // remove conn from n.channels[ch]
    n.refs[ch]--
    if n.refs[ch] == 0 { n.bus.Unsubscribe(ch) }
}

One bus subscription per node per channel. Internal fanout is O(subscribers on node).

5. Presence

Presence = "who is online in channel X right now". Cheap versions use Redis sorted sets keyed by channel, scored by last heartbeat timestamp:

ZADD presence:thread:thr_01HX7 <now_ms> <user_id>
ZREMRANGEBYSCORE presence:thread:thr_01HX7 0 <now_ms - 30_000>
ZRANGE presence:thread:thr_01HX7 0 -1  -- currently online

The node updates the user's score on every heartbeat. A separate janitor cleans stale entries every 30 s.

For large channels, do not broadcast "X is online" to everyone; push presence diffs or count-only updates.

6. Cross-Device Fanout

A user has N devices connected to possibly N different nodes. Publishing to user:{user_id} reaches all of them via the bus. No extra logic needed if channel naming is correct.

7. Sticky Sessions and Routing

Balance connections across nodes uniformly (consistent hashing by device_id) or randomly. Sticky sessions are not required if the bus fans out properly, but they help in two cases:

• Fewer bus subscriptions when all devices of a user land on the same node (hash by user_id).
• Reconnection warm-path -- the same node often still has recent history cached.

8. Capacity Planning

Rules of thumb per node (modern hardware, well-tuned):

• Node.js / Go / Rust: 50k–100k idle WebSocket connections per 2 vCPU / 4 GiB node.
• Memory is usually the first bottleneck (~20–40 KiB per connection including TLS state + buffers).
• Outbound bandwidth: plan for the 95th-percentile channel fanout, not the mean.

Scale by adding nodes behind the LB; the bus absorbs fanout.

9. Failure Modes

• Bus outage: connections remain open but events stop flowing. Expose a health check that fails on bus disconnect so the LB drains the node.
• Thundering herd on deploy: stagger node restarts; add a startup warm-up delay before re-subscribing clients.
• Zombie connections: heartbeat timeout + OS-level TCP keepalive must both be set, otherwise half-open connections eat capacity.
• Redis Pub/Sub is not durable: if an event must survive subscriber absence, use Redis Streams with XADD + consumer groups, not PUBLISH.

10. Observability

Per node: open connections, messages/sec (in/out), bus lag, per-channel fanout p95, CPU %, memory. Alert on concurrent-connection drops > 10% minute-over-minute (likely bug or bus issue, not user behavior).

Checklist

• Each node maintains local subscription map with refcounted bus subscriptions.
• Pub/sub bus chosen (Redis, NATS, Kafka) with documented rationale.
• Channel naming is hierarchical and coarse; no per-connection channels.
• Presence is sorted-set + heartbeat with diff broadcasts for large channels.
• Routing strategy (random / hash-by-user / sticky) documented.
• Capacity limits per node measured, not guessed.
• Health check fails on bus disconnect to drain unhealthy nodes.
• Durable bus (Streams/Kafka) used where events must not be dropped.