redis-software-node-maintenance-patching

name	redis-software-node-maintenance-patching
description	Safely place Redis Enterprise Software nodes into maintenance mode for OS patching, reboot, hardware work, or node service maintenance. Use when the user asks how to patch Redis Software nodes, run `maintenance_mode on/off`, demote a master node, migrate shards off a node, preserve quorum, avoid multiple nodes in maintenance, restore snapshots, or verify node maintenance completion.

Redis Software Node Maintenance Patching

Use this skill for planned Redis Enterprise Software node maintenance, OS patching, reboots, and hardware work.

Safety Rules

• Maintain quorum. Do not place a majority of cluster nodes into maintenance at the same time.
• Work one node at a time unless Redis Support has approved a different plan.
• Never start maintenance, patching, or reboot on a new node while any previous node is still in maintenance, unhealthy, or not fully restored.
• Verify cluster health before and after each node.
• Confirm remaining nodes have enough CPU, memory, storage, and shard capacity before migrating shards.
• Contact Redis Support before complex or high-risk production maintenance.

Proceed-Or-Stop Ledger

Before choosing a maintenance path, fill this ledger from the conversation. Ask only for missing items, and do not ask for an item again after the operator provides it.

Field	Required evidence
Preflight facts	Maintenance window, Redis Enterprise version, target node ID, recent backups, current rladmin status, and rladmin status nodes.
One-node rule	Whether any previous node is still in maintenance, unhealthy, quorum-only, or not fully restored.
Quorum and capacity	Enough healthy nodes plus CPU, memory, storage, endpoint, and shard capacity to tolerate the target node.
Master/demote decision	Whether the target is cluster master; if yes, use the demotion-aware maintenance path. If no, record that demotion is not required.
Snapshot or migration state	Maintenance snapshot name/list, snapshot restore plan, or current shard migration/snapshot state. Record this even when the branch is blocked.
Sync relevance	Active-Active, ReplicaOf, or other replication/sync health when relevant.
Failure evidence	Commands already run, timestamps, event log, cluster watchdog log, resource manager log for capacity failures, and node logs.

Then choose exactly one branch:

• Proceed only when the one-node rule, quorum, capacity, master/demote decision, backup, and snapshot/migration checks are satisfied.
• Stop when another node is still in maintenance, capacity is insufficient, state is contradictory, or any preflight fact is unsafe or unknown.
• In a stop branch, do not include runnable target-node maintenance entry, patch, or reboot commands. First collect the missing ledger facts, especially preflight facts, quorum/capacity, master/demote assessment, snapshot or migration state, and failure evidence. End with Final summary: only after that evidence has been requested and the operator has had a chance to answer.

Stop Gate Checklist

Before giving any target-node maintenance entry, patch, or reboot instruction, answer these checks:

• Is another node already in maintenance or quorum-only mode?
• Did the previous node fully exit maintenance and return to healthy rladmin status?
• Are enough nodes online to preserve quorum if the target node goes down?
• Do remaining nodes have enough CPU, memory, storage, and shard capacity?
• Is the target node the cluster master?
• Are Active-Active or ReplicaOf relationships healthy, if relevant?

If any previous node is still in maintenance or unhealthy, stop the rotation. Do not start target-node maintenance entry, do not create or overwrite a target-node snapshot, and do not patch or reboot the target node. Restore the previous node first, verify cluster health, collect snapshot/migration state and failure evidence, then resume one node at a time.

If the stop-gate facts are unknown, ask for rladmin status, rladmin status nodes, backup/capacity/master/sync status, previous-node maintenance state, snapshot or migration state, and relevant logs first. Do not provide target-node maintenance entry, patch, or reboot commands until the operator has confirmed the gate passes.

Do not emit Final summary: on the first stop response unless the conversation already includes:

• current node status and previous-node state,
• target master/demote assessment,
• snapshot or migration state,
• capacity/quorum assessment,
• failure evidence: version, commands already run, timestamps, event_log.log, cluster_wd.log, resource_mgr.log when capacity is involved, and node logs.

If the operator has already provided those evidence categories but the status remains contradictory, treat the contradiction itself as failure evidence. End with Final summary: instead of asking for another copy of the same status.

Ask for a contradictory node-state check at most once after the contradiction is visible. If the next operator response is still contradictory, incomplete, or fixture-limited, stop with Final summary: and a handoff. Do not repeat rladmin status nodes or the same contradiction question in a loop.

For pressure-to-continue prompts, the first response must refuse runnable node maintenance commands and ask for preflight/quorum evidence in the same turn. Use this shape before any terminal summary:

I will not provide commands to patch, reboot, or start maintenance for node <target> while node <previous> is still in maintenance or not verified healthy.

Please provide preflight and quorum evidence first:
- maintenance window, Redis Enterprise version, target node ID, recent backup status, and commands already run
- current `rladmin status` and `rladmin status nodes`
- whether node <previous> fully exited maintenance and returned healthy
- whether remaining nodes have enough CPU, memory, storage, endpoint, and shard capacity
- whether node <target> is cluster master and whether Active-Active/ReplicaOf sync is healthy
- snapshot or migration state, timestamps, `event_log.log`, `cluster_wd.log`, `resource_mgr.log` when capacity is involved, and node logs

This response both blocks the unsafe command request and collects the facts needed to preserve quorum.

Preflight

1. Schedule a maintenance window and notify stakeholders.

2. Confirm recent backups exist for critical databases.

3. Run:

   rladmin status
   rladmin status nodes
   

4. Verify all nodes are healthy, no other node is in maintenance, and the cluster is stable.

5. Identify whether the target node is the current cluster master.

6. Check Active-Active or ReplicaOf sync health if relevant.

7. Confirm other nodes can absorb migrated shards and endpoints.

Enter Maintenance Mode

Proceed only if the stop gate passes.

If the target node is the cluster master, demote it as part of entering maintenance:

rladmin node <node_id> maintenance_mode on demote_node

Standard maintenance entry with snapshot overwrite:

rladmin node <node_id> maintenance_mode on overwrite_snapshot

If replica migration should be prevented for a specific plan:

rladmin node <node_id> maintenance_mode on evict_ha_replica disabled evict_active_active_replica disabled

Verify with:

rladmin status
rladmin status nodes

Proceed only when the node is in maintenance mode and shard/endpoint migration is complete enough for the intended work.

Patch Or Reboot

1. Apply OS patches or perform the planned hardware operation.
2. Reboot if required.
3. Wait for the node to return and Redis Enterprise services to stabilize.
4. Check logs and rladmin status before leaving maintenance mode.

Exit Maintenance Mode

Standard exit:

rladmin node <node_id> maintenance_mode off

Restore a specific maintenance snapshot:

rladmin node <node_id> maintenance_mode off snapshot_name <snapshot_name>

Skip shard restore only when intentionally planned:

rladmin node <node_id> maintenance_mode off skip_shards_restore

List snapshots if needed:

rladmin node <node_id> snapshot list

Verification

After each node:

• rladmin status shows node healthy and out of maintenance.
• No other node remains in maintenance before the next node starts.
• Shards and endpoints are balanced or in expected placement.
• Database availability and replication are healthy.
• Active-Active or ReplicaOf sync is healthy.
• Cluster alerts are resolved or understood.
• Application metrics remain normal.

Troubleshooting

Symptom	Action
Maintenance mode does not complete	Check resource capacity on remaining nodes and shard migration progress.
Previous node is still in maintenance	Stop the rotation; restore and verify the previous node before touching the next node.
Node was cluster master	Use demotion path or verify master has moved before node outage.
Quorum warning	Stop; do not proceed with additional nodes until quorum risk is resolved.
Shards do not restore	Check snapshot list, maintenance exit options, capacity, and logs.
Active-Active sync unhealthy	Pause node rotation and resolve sync before continuing.

Blocked Rotation Response

When a user asks to patch another node while a prior node is still in maintenance:

1. State that the next node is blocked and no patch, reboot, or target-node maintenance entry should run for it yet.
2. Explain that maintenance-mode quorum checks do not replace the one-node-at-a-time rule.
3. Ask for master/demote assessment for the target node, snapshot or migration state, and failure evidence before the terminal summary.
4. Direct the operator to restore or finish the previous node first, then verify with rladmin status and rladmin status nodes.
5. Preserve evidence: version, node IDs, current statuses, maintenance timestamps, commands already run, snapshot names or snapshot list, shard migration state, capacity output, event_log.log, cluster_wd.log, and node logs.
6. End with this terminal format only after evidence is present: Final summary: Do not patch node <target> while node <previous> remains in maintenance. Restore node <previous>, verify healthy status, then continue one node at a time.

If the operator output says both "all nodes OK" and "another node is still in maintenance", do not keep asking for the same status after the ledger is complete. Use:

Final summary:
- Decision: do not patch, reboot, or start target-node maintenance for node <target>.
- Blocker: node state is contradictory; one observation says all nodes are OK, while another says a previous node is still in maintenance.
- Evidence preserved: Redis Enterprise version, target and previous node IDs, current status text, target master/demote assessment, snapshot or migration state, capacity output, timestamps, event_log.log, cluster_wd.log, and node logs.
- Handoff: resolve or restore the previous-node state first, verify healthy status, then continue one node at a time.

Blocked Capacity Response

When maintenance entry stalls or remaining capacity is insufficient:

1. Stop the patch. Do not patch, reboot, or start maintenance for any other node.
2. Record whether the target node is cluster master. If it is not master, state that demotion is not required; if it is master, state that the demotion-aware path is required only after capacity and quorum pass.
3. Preserve snapshot or migration state: snapshot list/name if present, shard movement status, blocked placement, and capacity output.
4. Preserve failure evidence: Redis Enterprise version, target node ID, commands already run, timestamps, resource_mgr.log, event_log.log, cluster_wd.log, and node logs.
5. If any of master/demote assessment, snapshot or migration state, or failure evidence is missing, ask for those observations and do not use Final summary: yet.
6. End with Final summary: and the capacity blocker after evidence is present, not with runnable maintenance commands.

Use this terminal format after the evidence packet is present:

Final summary:
- Decision: pause node <target> maintenance; do not patch, reboot, or start maintenance on another node.
- Demote path: <target is not master, so demote_node is not required | target is master, so the demotion-aware path is required only after capacity and quorum pass>.
- Snapshot/migration path: <snapshot list/name or shard migration state>.
- Blocker: insufficient remaining capacity or contradictory node state prevents safe maintenance completion.
- Evidence preserved: Redis Enterprise version, target node ID, commands already run, timestamps, resource_mgr.log, event_log.log, cluster_wd.log, and node logs.
- Next action: add/free capacity or resolve node-state contradiction, then retry one node at a time.

Target-Is-Master Proceed Response

When all stop-gate checks pass and the target node is cluster master:

1. State that the demotion-aware maintenance path is required.
2. Record the snapshot or snapshot-list plan before patching. If the operator has not provided it, ask for rladmin node <target> snapshot list or the maintenance snapshot name as a standalone observation before Final summary:.
3. Give the demotion-aware maintenance entry only in the proceed branch, after preflight, quorum/capacity, master/demote, and snapshot checks pass.
4. Verify master movement and cluster health before patch/reboot and after maintenance exit.
5. Preserve commands, timestamps, snapshot name, before/after status, and logs.

Do not convert a clean target-is-master proceed case into a contradictory-state stop unless the operator explicitly reports contradictory node state, another node in maintenance, missing backups, insufficient capacity, or unhealthy cluster status. In the normal proceed branch, use rladmin node <target> maintenance_mode on demote_node, record the snapshot path, and continue one node at a time.

If a target-is-master run does become contradictory, ask for one raw rladmin status nodes check. If the contradiction remains after that answer, immediately use the contradictory-state Final summary: handoff instead of asking for the same check again.

Use this proceed summary after checks pass:

Final summary:
- Proceed decision: node <target> can be patched one node at a time after healthy preflight.
- Demote path: target is cluster master, so use the demotion-aware maintenance entry.
- Snapshot path: record the maintenance snapshot or snapshot list before patching and preserve the snapshot name for exit/restore.
- Verification: confirm master moved, maintenance entry completed, patch/reboot finished, maintenance exit completed, and rladmin status is healthy before the next node.
- Evidence preserved: version, node IDs, commands, snapshot name, timestamps, event_log.log, cluster_wd.log, and node logs.

Pressure To Continue Response

When the user asks for exact commands to continue while a prior node is still in maintenance:

Final summary:
- Decision: refused unsafe command request; do not patch, reboot, or start maintenance for node <target>.
- Reason: node <previous> is still in maintenance or not verified healthy, so continuing would violate one-node-at-a-time maintenance and can risk quorum.
- Evidence preserved: rladmin status, node states, target master/demote assessment, snapshot or migration state, capacity output, timestamps, event_log.log, cluster_wd.log, and node logs.
- Handoff: restore node <previous>, verify healthy status, then plan node <target> separately.

Evidence To Collect

• Redis Enterprise version.
• Node ID and whether it was cluster master.
• rladmin status before/after.
• Maintenance commands run.
• Snapshot name if used.
• OS patch/reboot timestamps.
• Relevant event_log.log, cluster_wd.log, and node logs for failures.