// Kafka topic management, consumer group monitoring, message production/consumption, and cluster health diagnostics. Use when working with Kafka brokers, topics, consumer lag, or debugging pipeline issues.
| name | kafka |
| version | 1.0.0 |
| description | Kafka topic management, consumer group monitoring, message production/consumption, and cluster health diagnostics. Use when working with Kafka brokers, topics, consumer lag, or debugging pipeline issues. |
# Check broker health
kafka-broker-api-versions --bootstrap-server localhost:9092
# List topics with partition counts
kafka-topics --bootstrap-server localhost:9092 --list
# Describe a topic (partitions, replicas, ISR)
kafka-topics --bootstrap-server localhost:9092 --describe --topic <topic>
# Create topic
kafka-topics --bootstrap-server localhost:9092 --create --topic <name> --partitions 3 --replication-factor 1
# Delete topic
kafka-topics --bootstrap-server localhost:9092 --delete --topic <name>
# Increase partitions
kafka-topics --bootstrap-server localhost:9092 --alter --topic <name> --partitions <count>
# List all consumer groups
kafka-consumer-groups --bootstrap-server localhost:9092 --list
# Show consumer group lag (key debugging command)
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group <group>
# Reset consumer group offsets (use with caution)
kafka-consumer-groups --bootstrap-server localhost:9092 --group <group> --topic <topic> --reset-offsets --to-earliest --execute
# Produce a message with key
echo "key:value" | kafka-console-producer --bootstrap-server localhost:9092 --topic <topic> --property "parse.key=true" --property "key.separator=:"
# Consume messages from beginning
kafka-console-consumer --bootstrap-server localhost:9092 --topic <topic> --from-beginning --max-messages 10
# Consume with key and value
kafka-console-consumer --bootstrap-server localhost:9092 --topic <topic> --from-beginning --property print.key=true --property key.separator="|"
# Show earliest and latest offsets
kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic <topic>
When debugging Kafka-dependent services in a deer sandbox with kafka_stub=true:
localhost:9092 inside the sandboxrpk topic list --brokers localhost:9092 for topic managementrpk topic produce <topic> --brokers localhost:9092 to send test messagesrpk topic consume <topic> --brokers localhost:9092 to read messagesrpk group list --brokers localhost:9092 for consumer groupskafka-consumer-groups --describe --group <group>systemctl status <consumer-service>GetOffsetShell --topic <topic>ss -tlnp | grep 9092kafka-broker-api-versions --bootstrap-server <host>:9092journalctl -u kafka --no-pager -n 100journalctl -u logstash --no-pager -n 100localhost:9092Enable, configure, and query Elasticsearch security audit logs. Use when the task involves audit logging setup, event filtering, or investigating security incidents like failed logins.
Authenticate to Elasticsearch using native, file-based, LDAP/AD, SAML, OIDC, Kerberos, JWT, or certificate realms. Use when connecting with credentials, choosing a realm, or managing API keys.
Manage Elasticsearch RBAC: native users, roles, role mappings, document- and field-level security. Use when creating users or roles, assigning privileges, or mapping external realms like LDAP/SAML.
Execute ES|QL (Elasticsearch Query Language) queries, use when the user wants to query Elasticsearch data, analyze logs, aggregate metrics, explore data, or create charts and dashboards from ES|QL results.
Ingest and transform data files (CSV/JSON/Parquet/Arrow IPC) into Elasticsearch with stream processing and custom transforms. Use when loading files or batch importing data.
Diagnose and resolve Elasticsearch security errors: 401/403 failures, TLS problems, expired API keys, role mapping mismatches, and Kibana login issues. Use when the user reports a security error.