agentic-engineering-for-apache-kafka

Generate a TestContainers Java test class that spins up ShadowTraffic in-process to populate a Kafka topic with synthetic data during tests. Invokes the kafka-shadowtraffic skill to build the ShadowTraffic config, then adapts it for a containerized test network and scaffolds a JUnit 5 test class with Kafka, optional Schema Registry, and ShadowTraffic containers wired together. Use when the user says "set up a Java TestContainers test with ShadowTraffic", "write a Java test that generates Kafka data", "create a TestContainers test for my Kafka consumer", "I want to test my Kafka consumer with synthetic data in Java", or "scaffold a ShadowTraffic Java test". Do NOT use for non-Java projects, standalone Docker setups (use kafka-shadowtraffic instead), or creating Kafka topics.

Generate a ShadowTraffic configuration to populate a Kafka topic with realistic synthetic data. Discovers the target topic, its key and value schemas, and the correct serializers from the live cluster via any attached Kafka MCP server, then writes a ready-to-run `shadowtraffic-config.json` and Docker command. Use when the user says "populate my Kafka topic with test data", "set up ShadowTraffic for topic X", "generate synthetic events into my topic", "I need fake data flowing into Kafka", "seed my topic with data", or "mock data for my Kafka topic". Do NOT use for creating topics, reviewing schemas, or building Kafka consumers.

Scaffold a production-ready Python Kafka producer and consumer using `confluent-kafka-python`, with Schema Registry, graceful shutdown, idempotent producer, tests and a complete project layout. Discovers the target topic, partition count and registered JSON Schema directly from the live cluster via any attached Kafka MCP server (Lenses MCP, Confluent's MCP, Aiven's, custom) before asking the user. Use when user says "build a Python Kafka consumer", "scaffold a Kafka client in Python", "add Kafka to my Python service", "consume from `<topic>` in Python" or "write a Kafka producer". Do NOT use for Kafka Connect connectors, schema-evolution reviews or non-Python clients.

Review Kafka Connect connector configurations for common misconfigurations using the Lenses MCP server. Checks error handling, DLQ setup, converters, transforms, task count and task health. Use when user says "review connectors", "check connector configs", "why is my connector failing" or asks about Kafka Connect configuration. Do NOT use for creating, deploying or controlling connectors.

Analyse Kafka consumer group lag using the Lenses MCP server. Diagnoses lag causes (throughput bottlenecks, rebalancing, partition skew, stalled consumers) and suggests remediation. Use when user says "check consumer lag", "why are consumers slow", "lag report" or asks about consumer group health or offset progress. Do NOT use for resetting offsets or managing consumer groups.

Review dead letter queue implementations for completeness using the Lenses MCP server. Checks DLQ topic existence, configuration, monitoring, metadata preservation, retry logic, reprocessing paths and connector DLQ alignment. Use when user says "review dead letter queues", "check DLQ setup", "DLQ audit" or asks about error handling, message failures or reprocessing. Do NOT use for reprocessing DLQ messages or managing consumer offsets.

Review Kafka producer and consumer performance configurations in both the live cluster (via Lenses MCP) and the codebase. Flags un-tuned defaults, anti-patterns and missing best practices. Use when user says "review Kafka performance", "check producer configs", "tune Kafka settings" or asks about throughput, batching or compression. Do NOT use for cluster sizing or capacity planning.

Review Kafka schema changes (Avro, Protobuf, JSON Schema) for compatibility and evolution best practices using the Lenses MCP server. Detects breaking changes, missing defaults, schema drift and naming issues. Use when user says "review schema changes", "check schema compatibility", "will this schema break consumers" or asks about schema evolution. Do NOT use for creating new schemas from scratch or registering them in the cluster.

Audit Kafka security configuration across the codebase and live cluster using the Lenses MCP server. Checks authentication (SASL), encryption (SSL/TLS), authorisation (ACLs), secrets management and environment tier mismatches. Use when user says "audit Kafka security", "check security config", "is my cluster secure" or asks about authentication, encryption or credentials. Do NOT use for configuring certificates, creating SASL users or setting up ACLs.

Audit all Kafka topic configurations against production best practices using the Lenses MCP server. Checks replication factor, retention, partitions, compaction, naming conventions, orphaned topics and missing metadata. Use when user says "audit my topics", "check topic configs", "topic health check" or asks about retention, replication or partition settings. Do NOT use for creating, deleting or modifying topics.