realtime-streaming-event-driven

# Real-Time Streaming Architectures and Streaming Databases This skill establishes patterns for designing and operating systems oriented toward continuous event streams, unified batch-stream processing, and real-time OLAP analytics with millisecond-scale latencies. --- ## ⚡ 1. Lambda vs Kappa vs Unification in a Streaming DB ``` ┌─────────────────────────────────────────────────────────────┐ │ Operational Sources (OLTP: Postgres, MySQL, APIs, IoT) │ └──────────────────────────────┬──────────────────────────────┘ │ CDC (Debezium) / Event Producer ┌──────────────────────────────▼──────────────────────────────┐ │ Distributed Event Log (Apache Kafka / Apache Pulsar) │ └──────────────────────────────┬──────────────────────────────┘ │ ┌──────────────────┴──────────────────┐ │ │ ┌───────────▼───────────┐ ┌───────────▼───────────┐ │ Stateful Processor │ │ Streaming OLAP DB │ │ (Apache Flink) │ │ (Apache Pinot / │ │ (Windows, Aggregation)│ │ ClickHouse) │ └───────────────────────┘ └───────────┬───────────┘ │ ┌───────────▼───────────┐ │ Dashboards & APIs │ │ (< 50ms Query Latency)│ └───────────────────────┘ ``` --- ## 📊 2. Change Data Capture (CDC) Patterns with Debezium - **Log-Based CDC**: Directly reads the database's transaction logs (WAL in PostgreSQL, Binlog in MySQL) without the overhead of polling `SELECT` queries. - **Outbox Pattern**: Atomic write to the `outbox` table within the same relational database transaction to guarantee that domain events are never lost during network failures.