Real-time data streaming expertise covering Kafka architecture, stream processing with Flink and Kafka Streams, exactly-once semantics, windowing strategies (tumbling, sliding, session), watermarks, late data handling, schema registry, consumer group management, and production deployment patterns.
Use when the user asks about streaming architect, streaming architect best practices, or needs guidance on streaming architect implementation.
Do NOT use when the user needs a different specialized skill or is asking about an unrelated technology domain.
Real-time data streaming expertise covering Kafka architecture, stream processing with Flink and Kafka Streams, exactly-once semantics, windowing strategies (tumbling, sliding, session), watermarks, late data handling, schema registry, consumer group management, and production deployment patterns.
Use when the user asks about streaming architect, streaming architect best practices, or needs guidance on streaming architect implementation.
Do NOT use when the user needs a different specialized skill or is asking about an unrelated technology domain.
Real-time streaming is the architecture of processing data as it arrives, rather than in batches. This skill covers the design, implementation, and operation of streaming systems that handle millions of events per second with low latency, high reliability, and exactly-once processing guarantees.
Event Time: |-------|-------|-------|-------|
t=0 t=10 t=20 t=30
Processing Time progresses ->
Watermark = "I believe all events with event_time <= W have arrived"
With watermark delay of 5 seconds:
When processing-time clock is at T, watermark = max(event_times_seen) - 5s
Late data = event arriving after watermark has passed its event_time
Handling Late Data
# Spark Structured Streaming: watermark + late data policyfrom pyspark.sql.functions import window, col
windowed_counts = (
events
.withWatermark("event_time", "10 minutes") # Allow 10 min late
.groupBy(
window("event_time", "5 minutes"),
"event_type"
)
.count()
)
# Output modes with watermarks:# ... (condensed) ...
query = windowed_counts.writeStream \
.outputMode("append") \
.format("parquet") \
.option("checkpointLocation", "/checkpoints/windowed") \
.start("/output/windowed_counts")
Aggregate -> Join -> Dedup
Example: Real-time dashboard
Input: page views
-> Window(5min tumbling)
-> GroupBy(page_url)
-> Count()
-> Output: page_view_counts per 5-min window
Example: Stream-stream join
Input A: orders
Input B: payments
-> Join on order_id within 1-hour window
-> Output: enriched order with payment status
Production Deployment Patterns
# Kubernetes deployment for Kafka Streams applicationapiVersion:apps/v1kind:Deploymentmetadata:name:order-processorspec:replicas:6# Match partition count or be a divisorselector:matchLabels:app:order-processortemplate:spec:containers:-name:processor# ... (condensed) ...strategy:type:RollingUpdaterollingUpdate:maxSurge:1maxUnavailable:0# Zero-downtime deployment
Capacity Planning
Throughput Calculation
Producer throughput per broker:
Network: min(network_bandwidth, disk_write_speed) / replication_factor
Consumer throughput:
Limited by: network_bandwidth, consumer processing speed, partition count
Example:
10 GB/s network per broker
3 brokers, replication factor 3
Producer throughput: 10 GB/s / 3 (replication) = ~3.3 GB/s per broker
Total cluster throughput: ~10 GB/s (limited by replication)
With 12 partitions, 6 consumers:
Each consumer handles 2 partitions
Per-consumer throughput needed: total_throughput / 6
The question is about a fundamentally different technology domain
A more specific sibling skill covers the exact topic needed
The user needs a complete hands-on tutorial rather than expert guidance
Output Format
# Streaming Architect Analysis## Context Assessment
[Situation summary and constraints]
## Recommended Approach
[Primary recommendation with rationale]
## Implementation Steps1. [Step with specific details]
2. [Step with specific details]
3. [Step with specific details]
## Trade-offs and Considerations- [Key trade-off 1]
- [Key trade-off 2]
## Next Steps- [Immediate action item]
- [Follow-up action item]
Example
Input: "Help me implement streaming architect for a medium-scale production application"
Output: A structured analysis covering current state assessment, recommended streaming architect approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints.
Edge Cases
Legacy system integration: When streaming architect must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite
Scale mismatch: When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit
Team skill gaps: When the team lacks experience with the recommended approach, include learning resources and simpler alternatives
Conflicting requirements: When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities
