kafka-topic-audit

name	kafka-topic-audit
description	Audit all Kafka topic configurations against production best practices using the Lenses MCP server. Checks replication factor, retention, partitions, compaction, naming conventions, orphaned topics and missing metadata. Use when user says "audit my topics", "check topic configs", "topic health check" or asks about retention, replication or partition settings. Do NOT use for creating, deleting or modifying topics.
license	MIT
allowed-tools	Read, Grep, Glob, Bash, mcp__Lenses__check_environment_health, mcp__Lenses__list_topics, mcp__Lenses__get_topic, mcp__Lenses__get_topic_broker_configs, mcp__Lenses__get_topic_partitions, mcp__Lenses__list_topic_metadata, mcp__Lenses__list_consumer_groups_by_topic, mcp__Lenses__list_datasets
argument-hint	[required: environment name]
compatibility	Recommended - the Lenses MCP server (lenses-mcp) connected and configured with a valid environment. Any Kafka MCP that exposes an equivalent tool surface (list_topics, get_topic, etc.) will also work; without an MCP server, the skill falls back to codebase-only inspection and skips live-cluster checks.
metadata	{"author":"Tun Shwe","version":"1.0.0","mcp-server":"lenses-mcp","category":"mcp-enhancement","approach":"problem-first","patterns":"sequential-workflow, domain-intelligence","tags":["kafka","topics","audit","configuration","best-practices"]}

Kafka Topic Configuration Audit

Audits all topic configurations against production best practices. Misconfigured topics are the #1 cause of Kafka data loss - engineers create topics and forget to tune them.

Target environment: $ARGUMENTS

Workflow

Copy this checklist and track your progress:

Audit Progress:
- [ ] Step 1: Check environment health
- [ ] Step 2: Fetch all topics
- [ ] Step 3: Audit configurations against best practices
- [ ] Step 4: Check metadata completeness
- [ ] Step 5: Detect orphaned topics
- [ ] Step 6: Run consistency checks
- [ ] Step 7: Generate report

1. Check environment health for a high-level summary
2. Fetch all topics and their configurations
3. Audit each topic against best practices (see references/audit-rules.md)
4. Cross-reference metadata for completeness
5. Detect orphaned topics with no consumers
6. Report findings with prioritised recommendations

Step 1: Environment Overview

Use the Lenses MCP check_environment_health tool to get a quick summary:

• Broker count, topic count, consumer count, connector count
• Any existing issues flagged by Lenses

Expected output: Environment health summary with broker, topic and consumer counts.

Validation: If the environment is unhealthy or unreachable, stop and report the connection issue before proceeding.

Step 2: Fetch All Topics

Use the Lenses MCP list_topics tool to retrieve all topics with their configurations in one call.

For topics that need deeper inspection, use:

• get_topic for detailed config including partitions and consumers
• get_topic_broker_configs for broker-level config overrides
• get_topic_partitions for partition-level message counts and bytes

Expected output: Full list of topics with their configurations. If zero topics are returned, report this and stop.

Step 3: Audit Configurations

For each topic, check against the thresholds in references/audit-rules.md:

• Replication factor - RF=1 is critical, RF=2 is a warning in production
• Retention policies - unbounded growth, too short or excessively long
• Partition count - single-partition bottlenecks or excessive partitions
• Compaction settings - compact without keys, delete for state topics
• Naming conventions - must follow {domain}.{entity}.{event} pattern

Step 4: Metadata Completeness

Use the Lenses MCP list_topic_metadata tool to check:

• Topics missing descriptions
• Topics missing tags
• Topics without registered schemas (key or value)

Use list_datasets with filters (is_compacted, has_records) to find anomalies.

Step 5: Orphan Detection

For each topic, use list_consumer_groups_by_topic to check for active consumers.

• Warning: Topics with zero consumer groups (may be orphaned/dead)
• Suggestion: Topics with only inactive/empty consumer groups

Step 6: Consistency Checks

• Flag topics in the same domain with different retention policies
• Flag topics in the same domain with different replication factors
• Flag topics with inconsistent serialisation formats within a domain

Success Criteria

Quantitative

• Triggers on 90% of topic-related queries (test with 10-20 varied phrasings)
• Completes full audit in under 15 MCP tool calls
• 0 failed MCP calls per run

Qualitative

• Report is actionable without follow-up questions from the user
• Consistent severity categorisation (critical/warning/suggestion) across runs
• Every finding includes a concrete remediation step

Examples

Example 1: Routine weekly audit

User says: "Run a topic audit on the staging environment"

Actions:

1. Check staging environment health via Lenses MCP
2. Fetch all topics and configs
3. Audit each topic against rules in references/audit-rules.md
4. Check metadata completeness and orphaned topics Result: Full audit report with prioritised findings

Example 2: Pre-deployment check

User says: "Check if my topic configs are production-ready"

Actions:

1. Audit all topics for RF < 3, unbounded retention, single partitions
2. Flag any critical issues that would block a production deployment Result: Report highlighting critical issues that must be fixed before go-live

Example 3: Investigate a specific topic

User says: "Is the orders.payment.completed topic configured correctly?"

Actions:

1. Fetch detailed config for the specific topic using get_topic
2. Check broker-level overrides with get_topic_broker_configs
3. Verify metadata and consumer groups Result: Focused report on a single topic with all findings

Troubleshooting

Lenses MCP connection failed

Cause: Environment name is incorrect or Lenses agent is offline. Solution: Run check_environment_health first. Verify the environment name matches what list_environments returns.

No topics returned

Cause: Environment exists but has no topics or permissions are restricted. Solution: Confirm the cluster has topics via the Lenses UI. Check that the Lenses agent has read access.

Metadata endpoint returns empty

Cause: Schema Registry is not configured or topics have no registered schemas. Solution: This is a valid finding - report it as missing metadata rather than treating it as an error.

Output Format

## Topic Audit Report

### Environment: {name}
- Brokers: X | Topics: Y | Consumer groups: Z

### Critical (must fix)
- [topic-name] Description of the issue
  Current: {current value} | Recommended: {recommended value}

### Warning (should fix)
- [topic-name] Description of the issue
  Current: {current value} | Recommended: {recommended value}

### Suggestion (consider improving)
- [topic-name] Description of the issue
  Recommendation: How to fix it

### Summary
- X critical issues found
- Y warnings found
- Z suggestions found
- Topics audited: N
- Orphaned topics: M