multiplayer-concurrent-control

name

multiplayer-concurrent-control

description

Real-time multiplayer concurrency architecture - always-on invariants (version-aware undo, locking leases, socket preemption, optimistic UI, core/network isolation) plus scale-hardening topics deferred from MVPs (UDP channels, TURN, binary frames, delta patches, compression, sticky sessions, timing wheels) with the explicit triggers that graduate each one. Load when designing or extending multiplayer sync, or when deciding whether an MVP has outgrown WS-only JSON on one server instance.

Multiplayer Concurrent Control

Two-part contract for real-time multiplayer systems, distilled from the whiteboard multiplayer plan and its 10-pass adversarial review (~/work/whiteboard/.planning/multiplayer/; reference implementation: ~/work/whiteboard client + ~/work/whiteboard-server).

Part A invariants apply from day one — the MVP implements all of them. Part B topics are deliberately OUT of any locally-tested MVP: each enters only when its trigger fires, behind a config flag, with the simple path kept as fallback (the pattern: enableWebRTC: false + automatic WS fallback). A scale feature that your test environment cannot exercise is a liability, not an asset — if you add one, add the infrastructure that tests it (staging TURN, multi-instance compose, packet-loss simulation) in the same change.

---

Part A — Always-on invariants (MVP and beyond)

A1. Multiplayer Paradox Defense (version-aware undo)

Undo MUST be version-aware: an undo transaction aborts entirely if any target entity's current version drifted from the recorded one (a peer changed it) or the entity is locked. Expectations are per action type: undoing a DELETE expects the entity ABSENT; CREATE/UPDATE undos expect it present at the recorded version. Version-less (single-player) entities pass every present-check so local boards are never blocked.

A2. Remote/local history isolation

Remote-driven changes MUST NEVER enter the local undo stack or be re-broadcast as local edits. Two enforcement points, both required: (1) remote applies go through a partial-apply API that bypasses history and maintains its diff baseline; (2) history recording SKIPS entities currently remote-driven (locked, or carrying an interpolation target) — global input events (window-level mouseup) fire in every co-hosted client and will otherwise misattribute a peer's synced motion as a local change, echo it back, and version-stamp the peer's entity so THEIR undo aborts. (Found only by e2e; static review missed it.)

A3. Partial apply, never full-state reconcile

Applying one remote shape MUST NOT run a full-board load/reconcile path — differential loaders that remove absent entities will wipe the board on the first remote update. Full reconcile is reserved for file-load and the init flush (followed by a baseline reset).

A4. Socket Preemption (zombie socket fix)

On reconnect before the OS TCP timeout, the server MUST terminate the old socket registered to the same userId and accept the new one immediately — never reject as "already connected".

A5. Lock leases tied to real traffic

Locks expire on a short lease (~5s) refreshed by the holder's actual traffic (sync/update messages) — NEVER by a detached client-side setInterval, so Alt+Tab/sleep/crash releases locks organically. Locks also release on unlock and disconnect; grants/denials are explicit replies; non-holders' updates/unlocks are dropped server-side; current locks ride in the init payload.

A6. Optimistic UI with authoritative snap-back

Local interactions apply instantly (zero input lag) while the lock request races in the background. On rejection the client MUST abortInteraction() — destroy ghost overlays/DOM, cancel the gesture — and revert to the authoritative state (e.g. via the pending action's before snapshot).

A7. Refresh-rate independent interpolation

Never write remote sync coordinates directly to render state. Stage them in a target component and lerp with delta-time exponential decay (t = 1 − exp(−speed·dt)) so remote motion is physically identical at 30Hz and 144Hz. Only remote sync ever sets targets, so local gestures are never fought.

A8. Server-authoritative ordering and versions

The server stamps strictly incrementing per-room z-order sequence numbers on create (resuming from MAX+1 after restart) and owns the version counter (bumped per accepted update). Clients never invent versions.

A9. Server memory + payload hygiene

Hard-delete state (Map.delete / DELETE FROM) — no tombstones; stray updates for deleted ids are dropped. Cap frame size (maxPayload), clamp text lengths, scope every mutation query by room/board id (ids are client-supplied — an unscoped WHERE id = ? lets one room mutate another).

A10. Clean core isolation

The application core (ECS) knows nothing about networking. It exposes an API + event emitter; the emitter blocklists local-only singletons (camera, cursor, tool, selection) and its pause gating is REFCOUNTED so nested pause/resume pairs (a remote apply inside a read-only period) cannot accidentally re-enable emission.

A11. Reconnect discipline

Exponential backoff with jitter; on disconnect flip the client read-only (no doomed optimistic edits); every reconnect is a full re-init from a single init payload (state + locks in one message).

---

Part B — Scale graduation topics (OUT of MVP; add on trigger only)

B1. TCP/UDP protocol split (WebRTC data channels)

• Problem: TCP head-of-line blocking — one dropped packet stalls all queued sync frames.
• Shape: unordered/no-retransmit data channel for ephemeral idempotent state only (drag sync, cursors); critical state stays on the WS. Server side is SFU-style.
• Trigger: field reports of stuttery remote drags with measured packet loss (>~1%). Localhost shows zero loss — it can never justify this.
• Traps: native bindings (node-datachannel) fail to load on some platform/Node combos — lazy-require and degrade to WS-only with a warning. Always keep the N-second open-timeout fallback to WS: corporate NATs block UDP so often that the fallback IS the common path.

B2. STUN/TURN

Only meaningful with B1 in a real deployment. STUN is free; TURN relays cost bandwidth and need credential rotation. Cannot be validated locally AT ALL — needs staging plus a genuinely NATed client. "Works on localhost" is zero evidence.

B3. Binary sync frames

• Shape: [type u8][entity ref][Float64 coords…] ArrayBuffers. Use Float64 — JS numbers are f64 and far-from-origin coordinates on infinite canvases visibly quantize in f32.
• Honesty note: plain JSON already carries full f64 precision — bandwidth is the ONLY justification for binary, not precision.
• Trigger: measured bandwidth pressure (hundreds of concurrent viewers, mobile-data complaints). ~30 msg/s of geometry saves only ~5 KB/s per viewer.
• Trap: unreadable wire traffic; version every frame; keep a JSON debug path.

B4. Delta updates (JSON Patch, RFC 6902)

• Trigger: individual payloads >~10 KB (rich text, embedded data). For ~200-byte shapes the patch library + apply-order bug surface costs more than it saves.
• Prerequisite: end-to-end per-entity versions with full-payload resync on base mismatch — patches silently corrupt without it.

B5. Init-payload compression + cached serialization

• Bandwidth: compress the init message explicitly (zlib) at ~5-10x for JSON. Prefer that over permessage-deflate, which allocates a zlib context PER SOCKET (real memory at thousands of connections).
• CPU: keep a pre-serialized Buffer of master state (invalidated on write) so join storms don't re-serialize the board per joiner and block the event loop.
• Trigger: boards >~1000 shapes, or many users joining one room simultaneously.

B6. Sticky sessions / backplane (horizontal scaling)

• Problem: one room split across instances = room fragmentation.
• Shape: route by room id at the LB (cheap first step) OR go instance-agnostic with a pub/sub backplane (Redis) — pick one deliberately.
• Trigger: the moment a second server instance exists. Until then it is a deployment-doc note, not code.
• Trap: stickiness breaks on restart/scale-down — A11's full re-init on reconnect is what makes that survivable; never lose it.

B7. Timing wheels for lock expiry

One cancellable setTimeout per lock (reset on refresh) is already O(1) per event and survives to large scale — it is the correct MVP implementation, not a naive one. Never use a global polling sweep. Graduate to a hashed timing wheel (e.g. 512 slots, ~10ms tick) only when profiling shows timer churn dominating (millions of acquire/release cycles per minute).

B8. Presence fan-out batching (cursors)

• Problem: N users × 30 msg/s cursor broadcasts is N² fan-out (50 users ≈ 75k msg/s).
• Shape: server aggregates all cursor positions per room into ONE 10-20Hz frame; clients interpolate between frames (CSS transition or lerp). Escape remote-controlled names before they touch the DOM — the HTML presence overlay is the XSS surface canvas rendering avoids.
• Trigger: rooms >~10 concurrently active users.

---

Graduation checklist (copy into plans that defer Part B)

Topic	Trigger	Test infra required
B1 UDP channel	field packet-loss stutter	packet-loss sim / staging
B2 TURN	B1 + NATed users	staging + NATed client
B3 Binary frames	measured bandwidth pressure	frame-version tests
B4 JSON Patch	payloads >10 KB	version-gated apply tests
B5 Compression/caching	>1k shapes or join storms	load test
B6 Sticky sessions	>1 server instance	multi-instance compose
B7 Timing wheel	timer churn in profiles	lock-churn benchmark
B8 Cursor batching	>10 active users/room	fan-out load test