| license | Apache-2.0 |
| name | streaming-pipeline-architect |
| description | Kafka Streams, Flink, Spark Streaming, and CDC for real-time data pipelines. Activate on: streaming, Kafka Streams, Flink, Spark Streaming, CDC, Debezium, real-time pipeline, event stream processing. NOT for: message broker setup (use event-driven-architecture-expert), batch processing (use batch-processing-optimizer). |
| allowed-tools | Read,Write,Edit,Bash(npm:*,npx:*,docker:*,python:*) |
| category | Backend & Infrastructure |
| tags | ["streaming","kafka-streams","flink","cdc","real-time"] |
| pairs-with | [{"skill":"event-driven-architecture-expert","reason":"Streaming consumes from event-driven sources"},{"skill":"schema-evolution-manager","reason":"Stream schemas must evolve without breaking consumers"},{"skill":"lakehouse-architect","reason":"Streaming data lands in lakehouse tables"}] |
Streaming Pipeline Architect
Design and build real-time data pipelines using Kafka Streams, Apache Flink, Spark Structured Streaming, and Change Data Capture.
Activation Triggers
Activate on: "streaming pipeline", "Kafka Streams", "Flink", "Spark Streaming", "CDC", "Debezium", "real-time pipeline", "event stream processing", "stream-table join", "windowed aggregation"
NOT for: Message broker configuration → event-driven-architecture-expert | Batch ETL optimization → batch-processing-optimizer | Data warehouse loading → data-warehouse-optimizer
Quick Start
- Identify streaming use case — real-time dashboards, CDC replication, fraud detection, event enrichment
- Choose engine — Kafka Streams (simple, JVM), Flink (complex, stateful), Spark (unified batch+stream)
- Design CDC pipeline — Debezium captures DB changes → Kafka → stream processor → target
- Handle late data — configure watermarks and allowed lateness for windowed operations
- Plan checkpointing — enable exactly-once with Flink checkpoints or Kafka transactions
Core Capabilities
| Domain | Technologies |
|---|
| Stream Processing | Apache Flink 1.20+, Kafka Streams 3.8+, Spark Structured Streaming |
| CDC | Debezium 2.7+, Fivetran, Airbyte, Maxwell |
| Managed | Confluent Cloud, AWS Kinesis, GCP Dataflow |
| Connectors | Kafka Connect, Flink CDC connectors, Spark connectors |
| State | RocksDB (Flink/Kafka Streams), Delta Lake checkpoints |
Architecture Patterns
CDC Pipeline (Database → Kafka → Target)
PostgreSQL Debezium Kafka Stream Processor
┌──────────┐ ┌─────────────────┐ ┌──────────┐ ┌─────────────────┐
│ WAL │────→│ Debezium │────→│ Topics │────→│ Flink / KS │
│ (logical │ │ (Kafka Connect)│ │ per │ │ - Enrich │
│ repl.) │ │ │ │ table │ │ - Aggregate │
└──────────┘ └─────────────────┘ └──────────┘ │ - Transform │
└────────┬────────┘
│
┌────────────────────────┤
↓ ↓
Elasticsearch Snowflake / Delta
(search index) (analytics)
