rpk-topic

# rpk topic: Manage, Produce & Consume `rpk topic` is the `rpk` command group for all topic-level operations on a Redpanda cluster. It wraps the Kafka protocol to create, inspect, alter, and delete topics, and provides full-featured produce and consume commands with a powerful percent-escape format language. All subcommands read connection config from the active rpk profile or `-X` flags. ## Quickstart ```bash # 1. Create a 3-partition topic with replication factor 3 rpk topic create orders -p 3 -r 3 # 2. Create with configs set at creation time rpk topic create events -p 6 -r 3 \ -c cleanup.policy=compact \ -c retention.ms=86400000 # 3. Produce three keyed JSON records — key and value on each line (Ctrl-D to finish) # Note: when %k appears in --format, it reads the key from the input line and # takes precedence over any -k flag. Use one mechanism or the other, not both. rpk topic produce orders -f '%k %v{json}\n' order-1 {"id":1,"status":"pending"} order-2 {"id":2,"status":"shipped"} order-3 {"id":3,"status":"delivered"} # 4. Produce a single record inline via stdin redirect printf 'hello world\n' | rpk topic produce orders # 5. Consume from the beginning and stop at current end rpk topic consume orders -o start # 6. Consume last 5 records rpk topic consume orders -o -5 # 7. Consume from a specific offset until current end rpk topic consume orders -o 100:end # 8. Consume in a named consumer group rpk topic consume orders -g my-service # 9. Describe the topic (summary + configs by default) rpk topic describe orders # 10. Describe including partition detail rpk topic describe orders -p # 11. Alter a topic config rpk topic alter-config orders --set retention.ms=3600000 # 12. Delete a topic config key (revert to cluster default) rpk topic alter-config orders --delete retention.ms # 13. Add 2 more partitions rpk topic add-partitions orders -n 2 # 14. List all topics rpk topic list # 15. Delete a topic rpk topic delete orders ``` ## Subcommands | Subcommand | What it does | |---|---| | `create` | Create one or more topics with partitions, replicas, configs | | `list` (alias `ls`) | List topics with partition and replica counts | | `describe` (alias `info`) | Show topic summary, configs, and partition offsets | | `describe-storage` | Show tiered-storage cloud/local status per partition | | `alter-config` | Incrementally set, delete, append, or subtract config keys | | `add-partitions` | Add N new partitions to an existing topic | | `trim-prefix` | Move the log start offset forward, discarding old data | | `analyze` | Measure batch rate and batch size over a time window | | `delete` | Delete one or more topics | | `produce` | Write records from stdin to a topic | | `consume` | Read records from topics and print to stdout | ## create ```bash rpk topic create <TOPICS...> [flags] ``` | Flag | Type | Default | Description | |---|---|---|---| | `-p, --partitions` | int32 | `-1` | Number of partitions; `-1` uses `default_topic_partitions` | | `-r, --replicas` | int16 | `-1` | Replication factor (must be odd); `-1` uses `default_topic_replications` | | `-c, --topic-config` | string (repeatable) | | `key=value` config pair, e.g. `-c cleanup.policy=compact` | | `-d, --dry` | bool | false | Validate only; do not create | | `--if-not-exists` | bool | false | Skip silently if topic already exists | ```bash # Compact topic rpk topic create audit-log -p 12 -r 3 -c cleanup.policy=compact # Dry run rpk topic create test-topic -p 3 -r 3 -d ``` ## list ```bash rpk topic list [TOPICS...] [flags] ``` | Flag | Default | Description | |---|---|---| | `-d, --detailed` | false | Show per-partition leader/replica detail | | `-i, --internal` | false | Include internal topics (e.g. `__consumer_offsets`) | | `-r, --regex` | false | Parse topic names as regex | ```bash rpk topic list rpk topic list -r '^orders.*' rpk topic list -d ``` ## describe ```bash rpk topic describe <TOPICS...> [flags] ``` By default prints the `summary` and `configs` sections. Sections: | Flag | Description | |---|---| | `-s, --print-summary` | Print topic name, partitions, replicas | | `-c, --print-configs` | Print all config key/value/source pairs | | `-p, --print-partitions` | Print per-partition leader, epoch, replicas, offsets | | `-a, --print-all` | Print all three sections | | `--stable` | Add last-stable-offset column (for transactional topics) | | `-r, --regex` | Describe topics matching a regex | ```bash rpk topic describe orders -a rpk topic describe -r '^events.*' -a ``` ## alter-config ```bash rpk topic alter-config <TOPICS...> [flags] ``` Supports four operations (all repeatable): | Flag | Operation | |---|---| | `-s, --set key=value` | Set a config value | | `-d, --delete key` | Delete a key (revert to default) | | `--append key=value` | Append to a list-of-values key | | `--subtract key=value` | Remove from a list-of-values key | | `--dry` | Validate only; do not apply | | `--no-confirm` | Skip confirmation prompt | | `-r, --regex` | Parse the input topics as regular expressions and alter every matching topic | ```bash # Enable tiered storage on a topic rpk topic alter-config orders \ --set redpanda.remote.read=true \ --set redpanda.remote.write=true # Change retention rpk topic alter-config orders --set retention.ms=86400000 # Revert retention to cluster default rpk topic alter-config orders --delete retention.ms # Apply one change to every topic matching an expression rpk topic alter-config -r '^events-.*' --set retention.ms=86400000 ``` `-r/--regex` uses the same anchored expression format as `rpk topic list -r` and `rpk topic delete -r`, so list first to preview exactly which topics an expression will alter. ## add-partitions ```bash rpk topic add-partitions <TOPICS...> -n <N> [flags] ``` `-n` is required. Partitions can only be added, never removed. ```bash rpk topic add-partitions orders -n 6 ``` ## trim-prefix Moves the log start offset (low watermark) forward, deleting segments before the specified offset. Data before the new start offset is no longer readable. ```bash rpk topic trim-prefix [TOPIC] -o <OFFSET> [flags] ``` | Flag | Description | |---|---| | `-o, --offset` | Target offset (`47`, `end`, or `@<timestamp>`) | | `-p, --partitions` | Comma-separated partition list (default: all) | | `-f, --from-file` | File with topic/partition/offset rows | | `--no-confirm` | Skip the confirmation prompt | ```bash # Trim partition 0 to offset 1000 rpk topic trim-prefix orders -o 1000 -p 0 # Trim all partitions to a timestamp rpk topic trim-prefix orders -o "@2024-01-01T00:00:00Z" # Trim to the current end (delete everything) rpk topic trim-prefix orders -o end ``` ## analyze Consumes a time window of records and reports batch rate and size statistics. ```bash rpk topic analyze <TOPICS...> [flags] ``` | Flag | Default | Description | |---|---|---| | `-t, --time-range` | `-1m:end` | Time range to sample (e.g. `-24h:end`, `-48h:-24h`) | | `--batches` | `10` | Minimum number of batches to consume per partition | | `--timeout` | `10s` | How long to run | | `-a, --print-all` | false | Print all output sections | | `-s, --print-summary` | false | Global summary | | `--print-topics` | false | Per-topic summary | | `--print-partition-batch-rate` | false | Batch rate percentiles | | `--print-partition-batch-size` | false | Batch size percentiles | | `-r, --regex` | false | Parse topic names as regex | ```bash rpk topic analyze orders -t -1h:end -a rpk topic analyze -r '^orders.*' -t -24h:end -s --print-topics ``` ## describe-storage Requires the Admin API endpoint (from your profile or `-X admin.hosts=...`). Shows tiered storage cloud vs. local bytes, segment counts, offsets, and sync lag per partition. Not supported on Redpanda Cloud clusters (the Admin API is not exposed there); all other `rpk topic` subcommands are. ```bash rpk topic describe-storage <TOPIC> [flags] ``` | Flag | Description | |---|---| | `-a, --print-all` | All sections | | `-s, --print-summary` | Summary (name, mode, last-upload) | | `-z, --print-size` | Cloud + local bytes and segment counts | | `-y, --print-sync` | Upload lag, manifest sync | | `-o, --print-offset` | Cloud and local start/last offsets | | `-H, --human-readable` | Human-readable sizes/durations | ```bash rpk topic describe-storage orders -a -H ``` Cloud storage modes: `disabled`, `write_only`, `read_only`, `full`, `read_replica`, `cloud_topic`, `cloud_topic_read_replica`, `tiered_cloud_topic`. The three cloud-topic modes print the L0/L1 SIZE layout rather than the segment-based tiered-storage one; see [manage.md](references/manage.md) for the column sets. ## delete ```bash rpk topic delete <TOPICS...> [flags] ``` `-r, --regex` parses topic names as regular expressions. Expressions are anchored with `^` and `$`. ```bash rpk topic delete old-topic rpk topic delete -r '^test-.*' ``` ## produce See [Produce Reference](references/produce.md) for full detail. ```bash rpk topic produce [TOPIC] [flags] ``` Key flags: | Flag | Default | Description | |---|---|---| | `-f, --format` | `%v\n` | Input record format (percent-escape syntax) | | `-k, --key` | | Fixed key for all records (parsed `%k` in format takes precedence) | | `-H, --header` | | `key:value` header (repeatable) | | `-z, --compression` | `snappy` | `none`, `gzip`, `snappy`, `lz4`, `zstd` | | `--acks` | `-1` | `-1`=all ISR, `0`=none, `1`=leader | | `-p, --partition` | `-1` | Direct-produce to this partition | | `-Z, --tombstone` | false | Produce empty value as null (tombstone) | | `--schema-id` | | Schema ID or `topic` (TopicName strategy) for value | | `--schema-key-id` | | Schema ID or `topic` for key | | `--schema-type` | | Fully-qualified Protobuf message type for value | | `--schema-key-type` | | Fully-qualified Protobuf message type for key | | `--schema-context` | | Schema Registry context to resolve schemas in; unset uses the topic's `redpanda.schema.registry.context` | | `--allow-auto-topic-creation` | false | Auto-create the topic if it does not exist | | `-o, --output-format` | `Produced to partition %p at offset %o with timestamp %d.\n` | Line printed to stdout after each successful record | | `--delivery-timeout` | `0` | Per-record delivery timeout (min 1s) | | `--max-message-bytes` | `-1` | Max batch bytes before compression | ```bash # Fixed key, newline-delimited values from stdin rpk topic produce orders -k order-key # Key and value separated by space rpk topic produce orders -f '%k %v\n' # Add headers rpk topic produce orders -H source:backend -H env:prod # Encode with schema registry (TopicName strategy) rpk topic produce orders --schema-id=topic # Tombstone rpk topic produce orders -k deleted-key -Z ``` ## consume See [Consume Reference](references/consume.md) for full detail. ```bash rpk topic consume <TOPICS...> [flags] ``` Key flags: | Flag | Default | Description | |---|---|---|