redis-k8s-secret-auth-connection-failures

name	redis-k8s-secret-auth-connection-failures
description	Use when Redis Enterprise for Kubernetes applications fail to connect after Kubernetes Secret, REDB authentication, passwordless access, ACL/user, TLS certificate, service, DNS, or network policy changes, especially WRONGPASS, NOAUTH, NOPERM, auth failed, timeout, disconnect, stale Secret, application pods not restarted, or REDB/operator reconciliation issues.

Redis Kubernetes Secret Auth Connection Failures

Use this skill when Redis for Kubernetes connectivity breaks after a credential, Secret, TLS, or access-mode change. The main job is to prove whether the failure is caused by Redis database auth, the Kubernetes Secret source, the application reload path, or network/TLS routing.

Safety Rules

• Never ask for raw passwords, private keys, or complete connection strings with embedded secrets.
• Decode Secrets only in approved secure environments and do not paste decoded values into tickets or chat.
• Treat Kubernetes custom resources and GitOps manifests as the source of truth when reconciliation is active.
• Do not rotate credentials repeatedly until you know which workload is still using which Secret.
• Track each requested observation. Do not ask for the same namespace, REDB, deployment, Secret metadata, app error, or redis-cli result twice.

Step-Advance Ledger

Use this ledger every turn. Mark a step complete when the operator supplies the corresponding observation, then move forward. An unchecked item means collect that one item; a checked item should not be requested again.

Step	Complete when
Classify	Namespace, REC, REDB/REAADB, service, port, exact error family, recent change, credential source, and pod restart status are known.
Desired state	REDB/REAADB auth/TLS state, Secret metadata/key names, and application deployment env or mount references are inspected with values redacted.
Same-path test	redis-cli was run from the same namespace and network path, using redacted Secret/TLS material where needed.
Interpret	The observed error is mapped to stale pod, missing auth fields, wrong Secret source, TLS CA mismatch, ACL issue, or network/service failure.
Remediate	The justified restart/config/Secret/TLS fix is identified. Do not rotate credentials unless state checks prove rotation is needed.
Evidence	Redacted REDB, Secret metadata, deployment, pod age/restart, service/endpoints, app error, and redis-cli result are preserved.

After Interpret, Remediate, and Evidence are complete, preserve evidence as an explicit action before the terminal summary. Use the exact heading Evidence preserved with secret values redacted: followed by the evidence packet, then end with Final summary:. Do not ask for another exact error line or another copy of the same YAML.

When writing Final summary:, do not include new questions or unsupported checks. Use only diagnosis, evidence, remediation, and verification. Extra follow-up asks tend to create unknown actions after the necessary milestones are already complete.

Initial Classification

Collect:

• namespace, REC name, REDB or REAADB name, service name, and port
• exact error: WRONGPASS, NOAUTH, NOPERM, TLS failure, timeout, or disconnect
• recent change: Secret update, REDB auth change, passwordless toggle, TLS/cert update, service or network policy change, app deployment change
• how the application reads credentials: env var, mounted Secret, external secret manager, Helm values, config file, or sidecar
• whether the failing pod restarted after the Secret changed

Ask for these classification facts once. If the operator response includes only the error family (WRONGPASS, NOAUTH, TLS handshake failure, timeout, or disconnect), continue with the state checks; do not stall waiting for a longer error string unless the branch is still ambiguous after the same-path test.

Check Redis And Kubernetes Desired State

Inspect database and cluster health before focusing on the application:

kubectl get redb <database-name> -n <namespace> -o yaml
kubectl get rec -n <namespace>
kubectl get pods -n <namespace> -o wide
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Then identify the Secret references from both sides:

kubectl get secret <database-secret> -n <namespace> -o yaml
kubectl get deployment <app-deployment> -n <namespace> -o yaml
kubectl get pod <app-pod> -n <namespace> -o yaml

Verify:

• the REDB references the intended Secret name and namespace
• the Secret has the expected key shape for the installed operator and database spec
• the application reads the same intended credential source
• external-secret controllers or GitOps tools have reconciled the latest value
• database and application auth modes match: password-only, username/password, or passwordless

Restart Cached Clients

Kubernetes Secret updates do not guarantee a running process reloads credentials. Restart every Redis client workload that may cache auth or TLS material:

kubectl rollout restart deployment/<deployment-name> -n <namespace>
kubectl rollout status deployment/<deployment-name> -n <namespace>
kubectl get pods -n <namespace> -o wide

Include workers, sidecars, scheduled jobs, migration jobs, monitoring agents, and long-lived admin tools.

Only recommend restart after desired state and same-path testing justify stale client material. If the same-path test succeeds with current Secret/TLS material and the failing pods are older than that material, restart affected client workloads and then finish with a verification summary.

In every NOAUTH branch that recommends restarting clients, include the concrete rollout commands. Do not write only "restart affected client pods" or "restart workloads" in prose:

kubectl rollout restart deployment/<deployment-name> -n <namespace>
kubectl rollout status deployment/<deployment-name> -n <namespace>

Prove The Network Path

Run redis-cli from the same namespace and network policy scope as the failing workload. Prefer an application pod or a temporary debug pod with the same egress path.

Password-only auth:

redis-cli -h <service-name> -p <port> -a '<password>' PING

Username and password:

redis-cli -h <service-name> -p <port> --user <username> -a '<password>' PING

TLS:

redis-cli -h <service-name> -p <port> --tls --cacert /path/to/ca.pem --user <username> -a '<password>' PING

Interpretation:

• PONG: Redis, service, DNS, and network path are likely good; focus on application config, secret reload, URL encoding, client TLS flags, and stale pods.
• WRONGPASS: wrong value, wrong user mode, stale Secret, or Redis auth mode mismatch.
• NOAUTH: client is not sending credentials even though Redis requires them.
• NOPERM: auth succeeded, but ACL or role permissions are insufficient.
• timeout or disconnect: check Service, Endpoints, NetworkPolicy, DNS, TLS mode, and database health.

Terminal Branches

Use these branch templates as compact endings after the ledger is complete. Keep the wording concrete and avoid adding new asks below the final summary. Each branch must include the standalone evidence packet heading before Final summary: so the evidence preservation is not hidden inside prose.

WRONGPASS after Secret update

Use this branch when the REDB references the intended Secret, same-path redis-cli succeeds with the current Secret, and app pods predate the Secret update:

Evidence preserved with secret values redacted:
- REDB/REAADB: desired state, status, auth mode, and referenced Secret names.
- Secret metadata: name, namespace, key names, resource version or update time; no decoded values.
- Application: deployment env refs or mounts, pod age/restart time, and rollout history.
- Network path: service, endpoints, namespace, port, and same-path redis-cli result.
- Error: exact WRONGPASS text from app logs, redacted if needed.

Final summary:
- Diagnosis: stale Redis client pods are still using old auth material.
- Evidence: REDB desired state, Secret metadata/key names, deployment env or mounts, pod age, same-path redis-cli result, and exact WRONGPASS error were preserved with secret values redacted.
- Remediation: run `kubectl rollout restart deployment/<deployment-name> -n <namespace>` and `kubectl rollout status deployment/<deployment-name> -n <namespace>` for every Redis client workload that may cache credentials, including workers and long-lived tools, then verify new pods and PONG.

NOAUTH after passwordless disabled

Use this branch when unauthenticated redis-cli returns NOAUTH, authenticated redis-cli succeeds, and the application config is not sending credentials:

Evidence preserved with secret values redacted:
- REDB/REAADB: auth mode, status, and referenced Secret names.
- Application: deployment auth fields, env refs or mounts, and pod restart state.
- Network path: service, endpoints, namespace, port, unauthenticated redis-cli result, and authenticated redis-cli result.
- Error: exact NOAUTH text from app logs, redacted if needed.

Final summary:
- Diagnosis: Redis now requires auth, but the client is not sending credentials.
- Evidence: REDB auth mode, deployment auth fields, service/endpoint state, app error, and same-path unauthenticated/authenticated redis-cli results were preserved with secret values redacted.
- Remediation: restore the application's auth configuration or intentionally re-enable passwordless access, then run `kubectl rollout restart deployment/<deployment-name> -n <namespace>` and `kubectl rollout status deployment/<deployment-name> -n <namespace>` for affected client pods and verify recovery.

If the same-path unauthenticated test unexpectedly returns PONG, do not loop. Finish with:

Evidence preserved with secret values redacted:
- REDB/REAADB: desired state, status, auth/TLS fields, and referenced Secret names.
- Application: deployment credential references and pod restart state.
- Network path: service, endpoints, namespace, port, and same-path redis-cli result.
- Error: reported NOAUTH text from app logs, redacted if needed.

Final summary:
- Diagnosis: the tested Redis path does not reproduce the reported NOAUTH failure; it is either the wrong endpoint/path or auth is still effectively passwordless on that path.
- Evidence: REDB desired state, deployment credential references, service/endpoints, app NOAUTH report, and same-path redis-cli result were preserved with secret values redacted.
- Remediation: verify the application uses the same service/port tested, align auth configuration for that endpoint, then run `kubectl rollout restart deployment/<deployment-name> -n <namespace>` and `kubectl rollout status deployment/<deployment-name> -n <namespace>` for affected client pods.

TLS handshake failure after certificate rotation

Use this branch when redis-cli --tls succeeds with the current CA bundle but the application mounts or trusts old CA material:

Evidence preserved with secret values redacted:
- REDB/REAADB: TLS desired state, status, and referenced Secret names.
- Secret metadata: CA/client certificate Secret names, key names, resource versions, and update times; no private keys.
- Application: deployment mount paths, env refs, pod age/restart time, and rollout history.
- Network path: service, endpoints, namespace, port, and same-path redis-cli TLS result.
- Error: exact TLS handshake error from app logs, redacted if needed.

Final summary:
- Diagnosis: the application is using stale or mismatched TLS trust material.
- Evidence: REDB TLS state, Secret metadata, deployment mount paths, app TLS error, service/endpoints, and same-path redis-cli TLS result were preserved with private keys redacted.
- Remediation: mount the current CA/client TLS material, run `kubectl rollout restart deployment/<deployment-name> -n <namespace>` and `kubectl rollout status deployment/<deployment-name> -n <namespace>` for affected workloads, and verify same-path PONG plus application recovery.

Redacted Evidence Packet Template

Use this exact shape when preserving evidence:

Evidence preserved with secret values redacted:
- REDB/REAADB: name, namespace, status, auth/TLS fields, referenced Secret names.
- Secret metadata: name, namespace, key names, resource version, last update time; no decoded values.
- Application: deployment env refs or mount paths, pod age/restart time, rollout history.
- Network path: service, endpoints, namespace, port, DNS/network policy notes.
- Same-path test: redis-cli result from the app namespace/path, with password and private keys omitted.
- Error: exact WRONGPASS/NOAUTH/TLS/NOPERM/timeout text from app logs, redacted if needed.

Service, DNS, Network, And TLS Checks

kubectl get svc,endpoints -n <namespace>
kubectl describe svc <service-name> -n <namespace>
kubectl get networkpolicy -n <namespace>
kubectl logs <app-pod> -n <namespace>
kubectl logs deployment/<operator-deployment> -n <operator-namespace>

Check:

• service name and port in the application match the current database service
• endpoints exist and point to healthy Redis pods
• DNS resolves from the failing pod
• NetworkPolicy permits app-to-Redis traffic
• client TLS settings match the database TLS mode
• client trusts the correct CA and uses the expected SNI or hostname when required

Common Fix Paths

Symptom	Likely Cause	Next Action
WRONGPASS after Secret update	running pod still has old value	restart every client workload and verify pod age
WRONGPASS with new Secret	REDB and app read different Secret sources	align Secret references and GitOps/external-secret sources
NOAUTH	client no longer sends auth	restore auth fields or change database auth mode intentionally
passwordless mode fails	client still sends AUTH	remove username/password/auth config from the client
TLS handshake error	TLS mode or CA mismatch	update client TLS flags and mounted CA material, then restart client pods
timeout	service, endpoints, DNS, network policy, or database health	test from same namespace and inspect service/endpoints/events

Evidence To Preserve

Collect redacted artifacts:

• REDB or REAADB YAML and status
• referenced Secret name, namespace, key names, and last update time
• application deployment YAML showing env vars or mounts, with secret values removed
• pod restart time and rollout history
• service, endpoint, DNS, network policy, and TLS settings
• exact application/client error
• redis-cli result from the same network path
• REC status and operator logs when reconciliation looks unhealthy