Amazon MSK Provisioned operations, troubleshooting, and health assessment for Standard and Express brokers. Use whenever the user mentions Amazon MSK, MSK Provisioned, MSK Standard/Express brokers, Apache Kafka on AWS, `kafka.*` / `express.*` instance types, or the `AWS/Kafka` CloudWatch namespace. Covers MSK performance issues (high CPU, produce/fetch latency, TrafficShaping), consumer lag, storage/EBS issues, rolling restarts, Kafka version upgrades, SECURITY_PATCHING, BROKER_UPDATE, CloudWatch alarm design, Kafka client (producer/consumer) tuning, under-replicated partitions, unexpected broker reboots, and full MSK operational reviews / health checks / best-practices audits. Do NOT use for MSK Connect, MSK Serverless, or MSK Replicator. Do NOT use for authoring CloudFormation, CDK, or Terraform templates. Do NOT use for other AWS services (RDS, Aurora, S3, DynamoDB, Kinesis, Lambda, EC2) unless MSK is explicitly named.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Amazon MSK Provisioned operations, troubleshooting, and health assessment for Standard and Express brokers. Use whenever the user mentions Amazon MSK, MSK Provisioned, MSK Standard/Express brokers, Apache Kafka on AWS, `kafka.*` / `express.*` instance types, or the `AWS/Kafka` CloudWatch namespace. Covers MSK performance issues (high CPU, produce/fetch latency, TrafficShaping), consumer lag, storage/EBS issues, rolling restarts, Kafka version upgrades, SECURITY_PATCHING, BROKER_UPDATE, CloudWatch alarm design, Kafka client (producer/consumer) tuning, under-replicated partitions, unexpected broker reboots, and full MSK operational reviews / health checks / best-practices audits. Do NOT use for MSK Connect, MSK Serverless, or MSK Replicator. Do NOT use for authoring CloudFormation, CDK, or Terraform templates. Do NOT use for other AWS services (RDS, Aurora, S3, DynamoDB, Kinesis, Lambda, EC2) unless MSK is explicitly named.
Operate, troubleshoot, and assess Amazon MSK (Managed Streaming for Apache Kafka)
Provisioned clusters — both Standard and Express broker types. This skill covers
day-to-day operations (health assessments, monitoring setup) and ad-hoc incident
response (performance degradation, consumer lag, storage full, unexpected broker
reboots).
When to Use
Activate this skill when the user asks to:
Review, audit, or assess an MSK cluster for best practices, health, or
operational readiness.
Troubleshoot an MSK cluster problem: high CPU, high produce/fetch latency,
consumer lag, broker storage running out, TrafficShaping events, under-replicated
partitions, or an unexpected broker restart.
Set up MSK monitoring: choose a monitoring level, create recommended CloudWatch
alarms and dashboards, understand the metrics available in the AWS/Kafka
namespace.
Plan an MSK maintenance event: rolling restart, Kafka version upgrade, security
patching, broker instance type change.
Advise on Kafka client (producer / consumer) configuration when the client is
connecting to an MSK cluster.
Do not activate this skill for MSK Connect, MSK Serverless, or MSK Replicator
— those are separate services with their own operational surfaces.
Broker Type Determination
Determine the broker type first — many checks differ between Standard and Express.
Starts with kafka. (e.g. kafka.m5.large, kafka.m7g.xlarge) → Standard broker.
Starts with express. (e.g. express.m7g.large) → Express broker.
Key Standard vs Express differences
Standard brokers use customer-managed EBS volumes for storage. You choose
instance types (kafka.m5.*, kafka.m7g.*), provision EBS, and manage storage
scaling. Standard brokers have scheduled maintenance windows.
Express brokers provide fully managed, pay-as-you-go storage with no EBS
provisioning. Instance types are prefixed with express.m7g.*. Express brokers
offer up to 3× more throughput per broker than Standard, and have no maintenance
windows. Express enforces a fixed replication factor of 3 and
min.insync.replicas=2 — you cannot create topics with RF=1.
Critical Warnings
This skill is read-only. Every command in this file and in references/
that mutates cluster state — update-broker-storage, create-configuration,
update-cluster-configuration, update-monitoring, put-metric-alarm,
reboot-broker, and any partition reassignment — is a recommendation for
the operator to run after review. Present these as proposed remediations
with expected impact and preconditions; do NOT execute them, and do NOT
imply that the agent will run them.
NEVER reboot brokers while UnderReplicatedPartitions > 0 (Standard only —
Express brokers do not emit URP). This risks data loss and extended outages.
NEVER recommend partition reassignment without first checking replication
status. Reassignment during URP compounds the problem.
linger.ms=0 is the #1 cause of "high CPU" on MSK. ALWAYS check client
batch configuration before recommending broker scaling.
EBS throughput ceilings are invisible in Kafka metrics — ALWAYS check EBS
volume metrics (VolumeReadBytes, VolumeWriteBytes, VolumeQueueLength)
when diagnosing Standard broker latency.
Express brokers have NO customer-managed EBS — do NOT recommend EBS
expansion or provisioned EBS throughput for Express clusters.
Express brokers enforce fixed RF=3 and min.insync.replicas=2 — do NOT
attempt to create topics with RF=1 on Express. If RF=1 is needed, use Standard
brokers.
Quick Diagnostics
These five checks cover the most common MSK issues. Use them before loading a
reference file.
CpuUser + CpuSystem > 60%: Check RequestHandlerAvgIdlePercent
(PER_BROKER monitoring level). If < 30%, request threads are saturated. Check
client batch.size and linger.ms before recommending scaling.
KafkaDataLogsDiskUsed > 85% (Standard only): Recommend to the
operator that EBS be expanded via aws kafka update-broker-storage (do
not execute). Identify high-growth topics via per-topic BytesInPerSec to
size the increase. Express clusters use StorageUsed metric instead and
storage is fully managed.
UnderReplicatedPartitions > 0 (Standard only): Check if a maintenance
operation or broker restart is in progress. If URP is decreasing, wait for
recovery. Do NOT restart brokers or reassign partitions during URP. Express
brokers do not emit this metric — monitor ProduceThrottleTime,
FetchThrottleTime, and consumer lag instead.
Consumer OffsetLag / MaxOffsetLag increasing: Determine if broker-side
(high ProduceTotalTimeMsMean, CPU saturation) or client-side (slow
processing, insufficient consumers). Per-partition lag from
PER_TOPIC_PER_PARTITION monitoring level helps isolate hot partitions.
BytesInPerSec near throughput ceiling: For Standard, check EBS volume
type and calculate: BytesInPerSec × ReplicationFactor vs volume throughput
limit. For Express, check against the per-broker sustained performance limits
in the MSK quotas.
Which Reference Do You Need?
Route to a reference file based on the customer intent. Read the reference in
full before answering — do not paraphrase from memory.
Customer Intent
Reference
High CPU, high produce/fetch latency, slow cluster, TrafficShaping
references/troubleshoot-performance.md
Consumer lag increasing, rebalance storms, stuck consumer groups
references/troubleshoot-consumer-lag.md
Disk filling up, retention planning, tiered storage, EBS scaling
references/manage-storage.md
Setting up monitoring level, dashboards, recommended CloudWatch alarms
references/monitor-and-alarm.md
Rolling restart impact, patching, Kafka version upgrades, maintenance resilience
references/maintenance-operations.md
Producer / consumer configuration, IAM / SCRAM / TLS auth for clients
Use this workflow when the user asks for a review, audit, health check, or
assessment of an MSK cluster. The routing table above handles ad-hoc
troubleshooting; this section produces a consistent, comprehensive report.
Follow the steps in order for each target cluster. Do not skip steps. If a
step cannot be completed (e.g. a metric requires a higher monitoring level
than the cluster has enabled), record the gap in the report rather than
silently omitting the check.
Step 1 — Identify Target Clusters
Ask the user which MSK clusters to review. Accept any of:
Specific cluster names or ARNs and regions
"all clusters" in specific regions
"all MSK clusters in all regions"
If no scope is given, default to all configured account regions. Enumerate
clusters with aws kafka list-clusters-v2 per region.
Read ClusterInfo.Provisioned.BrokerNodeGroupInfo.InstanceType. Standard
brokers (kafka.*) and Express brokers (express.*) require different checks
in the later steps — some metrics only exist on one type.
Step 3 — Collect Cluster Configuration
For each cluster, gather:
aws kafka describe-cluster-v2 --cluster-arn <arn>
aws kafka list-nodes --cluster-arn <arn>
aws kafka get-bootstrap-brokers --cluster-arn <arn>
aws kafka list-cluster-operations-v2 --cluster-arn <arn> # last 30 days
aws kafka describe-configuration-revision \
--arn <configuration-arn> --revision <revision> # if a custom config is applied
Capture:
Cluster: state, Kafka version, number of broker nodes, AZ distribution
(ZoneIds), storage mode (EBS / Tiered), current version.
Logging:LoggingInfo.BrokerLogs (CloudWatch / S3 / Firehose destinations
and their Enabled flags).
Open monitoring:OpenMonitoring.Prometheus.JmxExporter.EnabledInBroker,
NodeExporter.EnabledInBroker.
Recent operations: From list-cluster-operations-v2, note any
SECURITY_PATCHING, BROKER_UPDATE, UPDATE_CLUSTER_CONFIGURATION,
UPDATE_STORAGE, or UPDATE_MONITORING events in the review window.
Step 4 — Detect Monitoring Level and Gaps
The EnhancedMonitoring value from Step 3 determines which checks are
available. At DEFAULT, most per-broker health metrics are still available
(CPU, disk, network, partitions, connections, memory, TrafficShaping), but the
following checks are not possible without upgrading:
VolumeQueueLength (EBS I/O queue depth — Standard only)
VolumeReadBytes / VolumeWriteBytes (EBS throughput utilization — Standard
only)
IAM connection metrics (IAMTooManyConnections)
Record the monitoring level and list any dimensions that will be scored
partially or skipped. Recommend upgrading to PER_BROKER if any dimension is
degraded by the current level.
Namespace: AWS/Kafka. Dimensions: Cluster Name and Broker ID for
per-broker metrics; Cluster Name and Consumer Group and Topic for
consumer lag; Cluster Name only for cluster-wide metrics.
Use one cloudwatch.GetMetricData batch per cluster where possible.
Period: 3600 (1 hour). StartTime: 7 days ago. EndTime: now.
Any alarm currently in ALARM state → HIGH (surface in report header).
7.4 Partition Health
PartitionCount per broker ≤ recommended limit for the broker instance
type. Above recommended but below max → MEDIUM. Above max → HIGH (blocks
update operations).
UnderReplicatedPartitions > 0 sustained (Standard only) → HIGH. Transient
during a SECURITY_PATCHING / BROKER_UPDATE operation from Step 3 →
INFO — do NOT flag.
Config-level ISR risk: topics with min.insync.replicas >= replication.factor
are a configuration error the CloudWatch metric will not surface. If the
user has provided topic configs, flag any such topic as HIGH.
Total per-broker throughput < 60% of baseline bandwidth (see
references/troubleshoot-performance.md for baseline table). 60-70% →
MEDIUM. > 70% → HIGH.
Create a cluster configuration (server.properties) — Operator-run (recommend, do not execute):
The --server-properties argument MUST be a real Kafka properties file with
one key=value per line, separated by actual newline characters — NOT the
literal two-character escape sequence \n. The MSK API accepts the bytes as-is;
if you pass "k1=v1\nk2=v2" as a single string with escaped newlines, MSK
stores ONE invalid property line and the cluster will fail to apply it.
Recommended pattern: write the properties to a local file with real newlines,
then pass it via fileb:// so the CLI uploads the raw bytes verbatim. Verify by
reading the revision back with describe-configuration-revision and
base64-decoding ServerProperties — you should see one property per line.
aws kafka update-broker-storage returns "storage is optimizing"
Previous storage expansion still in cool-down (minimum 6 hours)
Wait for optimization to complete. Check cluster state with describe-cluster-v2.
ClusterState is MAINTENANCE
Standard brokers undergoing patching. Express brokers stay ACTIVE during maintenance.
Wait for cluster to return to ACTIVE. Do not perform update operations during MAINTENANCE.
Consumer receives GROUP_COORDINATOR_NOT_AVAILABLE
Coordinator broker is temporarily unavailable during rolling restart or overloaded
Retry with backoff. Check if maintenance is in progress via list-cluster-operations-v2.
NotEnoughReplicasException on produce
Fewer brokers in ISR than min.insync.replicas (default: 2)
Check UnderReplicatedPartitions (Standard only). For Express, check ProduceThrottleTime and broker health instead — URP is not available. If a broker is down for maintenance, this is transient. Do NOT lower min.insync.replicas to work around this.
Severity Definitions (for review-style reports)
Severity
Definition
SLA
CRITICAL
Immediate risk to availability, security, or data integrity
