streaming-patterns
Provides Kafka producer and consumer best practices and dead letter queue patterns for streaming data; use when designing Kafka consumers or producers or implementing real-time pipelines.
Informations de source
- Dépôt
- sawrus/agent-guides
- Dernière activité de la source
- 16 octobre 2026 à 19:08
- Langue détectée de SKILL.md
- anglais
- Étoiles
- 19
- Forks
- 4
Options d'installation
Le prompt qui vérifie d'abord la source est sélectionné par défaut. Vous pouvez passer à une commande directe ou télécharger une copie locale.
Vérifiez les fichiers source
Lisez SKILL.md et les fichiers associés affichés par SkillsMP avant de décider de l'installer.
Affichage de SKILL.md
SKILL.md
Instructions source · Aperçu en lecture seule- name
- streaming-patterns
- description
- Provides Kafka producer and consumer best practices and dead letter queue patterns for streaming data; use when designing Kafka consumers or producers or implementing real-time pipelines.
# Skill: Streaming Data Patterns
## When to load
When designing Kafka consumers/producers or implementing real-time pipelines.
## Producer Best Practices
```python
producer = Producer({
"bootstrap.servers": settings.KAFKA_BROKERS,
"acks": "all", # Wait for all replicas
"retries": 5,
"enable.idempotence": True, # Exactly-once delivery
"compression.type": "snappy",
})
```
## Consumer Pattern
```python
consumer = Consumer({
"group.id": "order-processor-v1", # Version when changing logic
"auto.offset.reset": "earliest",
"enable.auto.commit": False, # Manual commit: only after successful processing
})
msg = consumer.poll(timeout=1.0)
try:
process(deserialize(msg.value()))
consumer.commit(msg) # Commit only after success
except ProcessingError as e:
publish_to_dlq(msg, e) # Dead-letter, don't block partition
consumer.commit(msg)
```
## Dead Letter Queue
```json
{
"original_topic": "orders.created.v1",
"original_offset": 12345,
"original_payload": "...",
"error_message": "Deserialization failed",
"error_timestamp": "2026-02-16T10:30:00Z",
"retry_count": 3
}
```
Voir sur GitHub