streaming-pipeline-architect

license	Apache-2.0
name	streaming-pipeline-architect
description	Kafka Streams, Flink, Spark Streaming, and CDC for real-time data pipelines. Activate on: streaming, Kafka Streams, Flink, Spark Streaming, CDC, Debezium, real-time pipeline, event stream processing. NOT for: message broker setup (use event-driven-architecture-expert), batch processing (use batch-processing-optimizer).
allowed-tools	Read,Write,Edit,Bash(npm:*,npx:*,docker:*,python:*)
category	Backend & Infrastructure
tags	["streaming","kafka-streams","flink","cdc","real-time"]
pairs-with	[{"skill":"event-driven-architecture-expert","reason":"Streaming consumes from event-driven sources"},{"skill":"schema-evolution-manager","reason":"Stream schemas must evolve without breaking consumers"},{"skill":"lakehouse-architect","reason":"Streaming data lands in lakehouse tables"}]

Streaming Pipeline Architect

Design and build real-time data pipelines using Kafka Streams, Apache Flink, Spark Structured Streaming, and Change Data Capture.

Activation Triggers

Activate on: "streaming pipeline", "Kafka Streams", "Flink", "Spark Streaming", "CDC", "Debezium", "real-time pipeline", "event stream processing", "stream-table join", "windowed aggregation"

NOT for: Message broker configuration → event-driven-architecture-expert | Batch ETL optimization → batch-processing-optimizer | Data warehouse loading → data-warehouse-optimizer

Quick Start

1. Identify streaming use case — real-time dashboards, CDC replication, fraud detection, event enrichment
2. Choose engine — Kafka Streams (simple, JVM), Flink (complex, stateful), Spark (unified batch+stream)
3. Design CDC pipeline — Debezium captures DB changes → Kafka → stream processor → target
4. Handle late data — configure watermarks and allowed lateness for windowed operations
5. Plan checkpointing — enable exactly-once with Flink checkpoints or Kafka transactions

Core Capabilities

Domain	Technologies
Stream Processing	Apache Flink 1.20+, Kafka Streams 3.8+, Spark Structured Streaming
CDC	Debezium 2.7+, Fivetran, Airbyte, Maxwell
Managed	Confluent Cloud, AWS Kinesis, GCP Dataflow
Connectors	Kafka Connect, Flink CDC connectors, Spark connectors
State	RocksDB (Flink/Kafka Streams), Delta Lake checkpoints

Architecture Patterns

CDC Pipeline (Database → Kafka → Target)

PostgreSQL              Debezium               Kafka              Stream Processor
┌──────────┐     ┌─────────────────┐     ┌──────────┐     ┌─────────────────┐
│  WAL     │────→│  Debezium       │────→│  Topics  │────→│  Flink / KS     │
│ (logical │     │  (Kafka Connect)│     │  per     │     │  - Enrich       │
│  repl.)  │     │                 │     │  table   │     │  - Aggregate    │
└──────────┘     └─────────────────┘     └──────────┘     │  - Transform    │
                                                           └────────┬────────┘
                                                                    │
                                           ┌────────────────────────┤
                                           ↓                        ↓
                                    Elasticsearch           Snowflake / Delta
                                    (search index)          (analytics)