backend-web-real-time

# Web Real-Time Communication ## Purpose Build real-time communication systems: WebRTC signaling, media server architecture (SFU/MCU), TURN/STUN infrastructure, live streaming, and real-time data channels. ## Agent Protocol ### Trigger User mentions WebRTC, real-time video/audio, media streaming, SFU, MCU, signaling server, TURN/STUN, live streaming, media server, coturn, or real-time communication infrastructure. ### Input Context - Use case (video call, live streaming, conferencing, data channel) - Expected participant count (1-1, small group <10, large group >10) - Network conditions (NAT/firewall, mobile, enterprise) - Media types (audio, video, screen share, data channel) - Deployment model (self-hosted, hybrid, cloud) - Latency requirements (real-time, near-real-time, low-latency streaming) ### Output Artifact Architecture design with signaling protocol, media server topology, TURN/STUN config, client integration code, and deployment infrastructure. ### Response Format No preamble. No postamble. No explanations. No filler/hedging/transitions. Strip articles a/an/the where unambiguous. Compress output. ### Completion Criteria - [ ] Signaling server protocol selected (WebSocket, HTTP, SSE) and implemented - [ ] Media server architecture chosen (SFU > MCU > P2P mesh) per scale requirements - [ ] TURN/STUN infrastructure specified with credentials management - [ ] ICE/STUN/TURN configuration working across target network topologies - [ ] Media negotiation (SDP offer/answer) flow implemented - [ ] Client SDK integration complete with reconnection logic - [ ] Bandwidth estimation and adaptation strategy defined - [ ] Recording/archiving strategy defined if required - [ ] Fallback for restricted networks documented ### Max Response Length 4096 tokens ## Architecture Decision Trees ### Topology Selection | Criterion | Mesh (P2P) | SFU (Selective Forwarding) | MCU (Multipoint Control) | |-----------|-----------|---------------------------|--------------------------| | Participants | 2-4 | 4-100+ | 4-50 | | Server bandwidth | None | O(n) — one uplink per peer | O(1) — single mixed stream | | Client bandwidth | O(n^2) — send to all peers | O(1) — send once, receive n-1 | O(1) — single stream | | Latency | Lowest | Low (no mixing) | Higher (mixing delay) | | CPU cost | Client-side | Server relays | Server transcodes | | Complexity | Simplest | Moderate | High | | Fallback | STUN only | STUN + TURN | STUN + TURN | Decision: Always prefer SFU for multi-party. Mesh for 1-1 calls. MCU only when legacy client compatibility required. ### Signaling Protocol Decision | Criterion | WebSocket | HTTP Long Polling | SSE | |-----------|-----------|-------------------|-----| | Latency | <50ms | 200-500ms | 100-300ms | | Bidirectional | Yes | Yes | Server→Client only | | NAT Traversal | Native ws:// wss:// | HTTP (always works) | HTTP (always works) | | Browser Support | All modern | All | All (except IE) | | Connection overhead | Upgrade handshake | New request per poll | Persistent connection | | Reconnection | Built-in (some libs) | Stateless | Built-in (EventSource) | Decision: WebSocket for production signaling. SSE for one-way broadcast. HTTP polling as last resort for restrictive networks. ### Codec Selection | Codec | Bitrate | Quality | Licensing | Browser Support | |-------|---------|---------|-----------|-----------------| | VP8 | 200-1500 kbps | Good | Royalty-free | Chrome, Firefox, Safari 14.1+ | | VP9 | 150-800 kbps | Better | Royalty-free | Chrome, Firefox | | H.264 | 200-1500 kbps | Excellent | Patent-encumbered | Chrome, Firefox, Safari, Edge | | AV1 | 100-500 kbps | Best | Royalty-free | Chrome (limited) | Decision: H.264 for widest compatibility. VP9/AV1 for bandwidth-constrained environments. ## Workflow ### WebRTC Architecture ``` Peer A → Signaling Server ← Peer B | | ↓ ↓ STUN/TURN Server ← → ICE Negotiation ← → STUN/TURN Server | | ↓ ↓ └─────── Media via SRTP/SCTP ────────┘ ``` ### Signaling Server Patterns | Protocol | When to Use | Example | |----------|-------------|---------| | WebSocket | Low latency, bidirectional | Socket.io, ws | | HTTP Long Polling | Simple, no WebSocket support | REST endpoints | | Server-Sent Events | One-way server→client | Event notifications | ### Media Server Architecture | Architecture | Description | Pros | Cons | |-------------|-------------|------|------| | Mesh (P2P) | Every peer connects to every other | No server cost | Bandwidth scales O(n^2) | | MCU | Server mixes all streams into one | Low client bandwidth | Single point of failure | | SFU | Server relays streams, selective forwarding | Scalable, flexible | Higher server bandwidth | ### SFU (Selective Forwarding Unit) - Recommended - Each peer sends one stream to SFU - SFU forwards to all other peers - Peers can choose which streams to receive - Scales to hundreds of participants ### TURN/STUN Server Setup | Component | Purpose | Service | |-----------|---------|---------| | STUN | Discover public IP/port for NAT traversal | coturn (self-hosted), Google STUN | | TURN | Relay media when P2P fails (NAT/firewall) | coturn (self-hosted), Twilio Network Traversal | ## Implementation Patterns ### Pattern: Signaling with WebSocket (Node.js) ```typescript // server/signaling.ts import { WebSocketServer, WebSocket } from 'ws'; interface SignalingMessage { type: 'offer' | 'answer' | 'ice-candidate' | 'join' | 'leave' | 'room-info'; roomId: string; senderId: string; payload: unknown; } const wss = new WebSocketServer({ port: 8080 }); const rooms = new Map<string, Map<string, WebSocket>>(); wss.on('connection', (ws) => { let userId: string; let currentRoom: string; ws.on('message', (data) => { const msg: SignalingMessage = JSON.parse(data.toString()); switch (msg.type) { case 'join': { userId = msg.senderId; currentRoom = msg.roomId; if (!rooms.has(msg.roomId)) rooms.set(msg.roomId, new Map()); rooms.get(msg.roomId)!.set(userId, ws); broadcastToRoom(msg.roomId, { type: 'room-info', roomId: msg.roomId, senderId: 'system', payload: { peers: Array.from(rooms.get(msg.roomId)!.keys()) } }, userId); break; } case 'offer': case 'answer': case 'ice-candidate': { const targetPeer = rooms.get(msg.roomId)?.get(msg.payload.targetId); if (targetPeer?.readyState === WebSocket.OPEN) targetPeer.send(JSON.stringify(msg)); break; } case 'leave': { rooms.get(msg.roomId)?.delete(userId); broadcastToRoom(msg.roomId, { type: 'leave', roomId: msg.roomId, senderId: userId, payload: {} }, userId); } } }); ws.on('close', () => { rooms.get(currentRoom)?.delete(userId); if (rooms.get(currentRoom)?.size === 0) rooms.delete(currentRoom); broadcastToRoom(currentRoom, { type: 'leave', roomId: currentRoom, senderId: userId, payload: {} }, userId); }); }); function broadcastToRoom(roomId: string, msg: SignalingMessage, excludeId?: string) { const room = rooms.get(roomId); if (!room) return; for (const [id, ws] of room) { if (id !== excludeId && ws.readyState === WebSocket.OPEN) ws.send(JSON.stringify(msg)); } } ``` ### Pattern: Client-Side WebRTC Peer Connection ```typescript // client/webrtc.ts export class WebRTCClient { private pc: RTCPeerConnection | null = null; private signaling: WebSocket; private localStream: MediaStream | null = null; private remoteStream = new MediaStream(); constructor(serverUrl: string, private userId: string, private roomId: string) { this.signaling = new WebSocket(serverUrl); this.setupSignaling(); } private setupSignaling() { this.signaling.onopen = () => { this.signaling.send(JSON.stringify({ type: 'join', roomId: this.roomId, senderId: this.userId, payload: {} })); }; this.signaling.onmessage = async (event) => { const msg = JSON.parse(event.data); switch (msg.type) { case 'offer': await this.handleOffer(msg); break; case 'answer': await this.pc!.setRemoteDescription(new RTCSessionDescription(msg.payload)); break; case 'ice-candidate': await this.pc!.addIceCandidate(new RTCIceCandidate(msg.payload)); break; } }; } async startCall(constraints: MediaStreamConstraints = { audio: true, video: true }) { this.localStream = await navigator.mediaDevices.getUserMedia(constraints); this.pc = new RTCPeerConnection({ iceServers: [ { urls: 'stun:stun.l.google.com:19302' }, { urls: 'turn:turn.example.com:3478', username: 'user', credential: 'pass' }, ], }); this.localStream.getTracks().forEach(t => this.pc!.addTrack(t, this.localStream!)); this.pc.ontrack = (event) => { event.streams[0].getTracks().forEach(t => this.remoteStream.addTrack(t)); }; this.pc.onicecandidate = (event) => { if (event.candidate) this.signaling.send(JSON.stringify({ type: 'ice-candidate', roomId: this.roomId, senderId: this.userId, payload: { targetId: 'peer-id', candidate: event.candidate } })); }; const offer = await this.pc.createOffer(); await this.pc.setLocalDescription(offer); this.signaling.send(JSON.stringify({ type: 'offer', roomId: this.roomId, senderId: this.userId, payload: { targetId: 'peer-id', sdp: offer } })); } private async handleOffer(msg: any) { this.pc = new RTCPeerConnection({ iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] }); this.pc.ontrack = (event) => { event.streams[0].getTracks().forEach(t => this.remoteStream.addTrack(t)); }; this.pc.onicecandidate = (event) => { if (event.candidate) this.signaling.send(JSON.stringify({ type: 'ice-candidate', roomId: this.roomId, senderId: this.userId, payload: { targetId: msg.senderId, candidate: event.candidate } })); }; await this.pc.setRemoteDescription(new RTCSessionDescription(msg.payload.sdp)); const answer = await this.pc.createAnswer(); await this.pc.setLocalDescription(answer); this.signaling.send(JSON.stringify({ type: 'answer', roomId: this.roomId, senderId: this.userId, payload: { targetId: msg.senderId, sdp: answer } })); } } ``` ### Pattern: SFU Selective Forwarding (mediasoup) ```typescript // sfu-server.ts import * as mediasoup from 'mediasoup'; import { WebSocketServer } from 'ws'; async function createSfu() { const worker = await mediasoup.createWorker(); const router = await worker.createRouter({ mediaCodecs: [ { kind: 'audio', mimeType: 'audio/opus', clockRate: 48000, channels: 2 }, { kind: 'video', mimeType: 'video/VP8', clockRate: 90000 }, { kind: 'video', mimeType: 'video/H264', clockRate: 90000, parameters: { 'level-asymmetry-allowed': 1, 'packetization-mode': 1, 'profile-level-id': '42e01f' } }, ], }); const wss = new WebSocketServer({ port: 8081 }); const peers = new Map<string, { transport: mediasoup.types.WebRtcTransport; producer: mediasoup.types.Producer | null }>(); wss.on('connection', (ws) => {