| name | lrctl-debug |
| description | Get ground truth about a LittleRed/Redis instance with the lrctl CLI when debugging. Use whenever investigating an e2e test failure, a stuck reconciliation, a suspected split-brain / ghost-master / ghost-node, a wrong master label, or any "what is the actual Redis/Sentinel/Cluster topology right now" question — instead of hand-rolling kubectl exec + redis-cli loops. Covers the read-only verbs: status, verify, inspect, debug-dump. |
lrctl-debug — ground truth for LittleRed instances
When debugging, do not hand-roll kubectl exec ... redis-cli loops to discover
topology. lrctl already gathers operator-side ground truth and cross-checks it.
Reach for it first.
lrctl is on PATH (installed via make install). If it's missing, build it:
make lrctl → ./bin/lrctl. It's also a kubectl plugin (kubectl lr ...) after
make install-plugin.
Global flags (apply to every verb)
-n <ns> / --namespace <ns> — target namespace (defaults to current context ns)-A / --all-namespaces — operate on every LittleRed in the cluster (no name arg)--json — machine-readable output; prefer this when parsing/reasoning, the
text form when showing a human--kubeconfig <path> — for e2e Kind clusters, point at the test kubeconfig--unmanaged --kind <standalone|sentinel|cluster> — inspect a Redis instance
that has no LittleRed CR (heuristic pod discovery); requires an explicit name
Name arg is optional for status/verify/inspect — omit it to act on all
instances in the namespace.
Which verb for which question
| Symptom / question | Verb | Why |
|---|
| "Is the CR healthy? what does the operator think?" | status | Reads CR .status only (Phase, Master, Ready counts, bootstrapRequired). Fast, no exec. |
| "Is the actual topology consistent? split-brain? ghost?" | verify | Gathers live ground truth, computes the authority master, flags ghosts/partitions, and prints recommended healing actions. This is the workhorse for failover bugs. |
| "Show me the raw redis-cli output from every pod" | inspect | Deep dive: per-pod INFO replication (sentinel) or CLUSTER NODES+CLUSTER INFO (cluster), plus SENTINEL MASTER from each sentinel. Use when verify flags something and you need the raw evidence. |
| "Capture everything for a bug report / later analysis" | debug-dump | Writes CR YAML, operator logs (last 10m), all pod logs (incl. previous for crashloops), live redis state, and k8s resources to a timestamped dir. Secrets auto-redacted. |
Recommended debugging order
lrctl status <name> -n <ns> — what the operator believes.lrctl verify <name> -n <ns> — what's actually true + heal actions. Start here
for any failover / master-label / consistency bug.lrctl inspect <name> -n <ns> — raw per-pod evidence when verify flags an issue.lrctl debug-dump <name> -n <ns> — when you want to preserve the whole state
(e.g. an e2e failure you can't reproduce on demand).
Reading verify output
- Sentinel mode: look at the Authority Master line.
[OK] Authority Master: <pod>
is good; GHOST(<ip>) means the consensus master IP has no live pod (the LR-008
ghost-master case); NONE means split-brain or uninitialized. [!] failover in progress and the Recommended Healing Actions list tell you what the operator
would do next.
- Cluster mode:
Cluster State, Total Slots Assigned: N / 16384 (anything <16384
= gaps), Ghost Nodes (in cluster but not in K8s → expect CLUSTER FORGET),
Network Partitions, and the per-master topology tree with replica link status.
e2e specifics
E2e tests run either against a throwaway Kind cluster (default make test-e2e) or
against an existing external cluster (make test-e2e SKIP_KIND_SETUP=true SKIP_OPERATOR_DEPLOY=true). Both are normal.
Which cluster does lrctl talk to?
- If the session context gives no explicit cluster/context, use the current
default kubectl context — just run
lrctl ... with no --kubeconfig. Don't
assume Kind.
- If the session context introduced an explicit context (a named kube-context, a
KUBECONFIG path, a Kind cluster name), use that one — pass --kubeconfig <path> or set the right context first.
When a test fails mid-run, lrctl verify -A is the fastest way to see which
instance is unhealthy across all the test namespaces at once.
A nonzero exit from verify means "not healthy/consistent" — useful as a signal,
but read the printed detail (or --json) to know why.
