| name | ops-investigation-k3s-upgrade-homelab |
| description | Instructions for updating the k3s homelab cluster one machine at a time while preserving custom systemd ExecStart arguments. |
| compatibility | opencode |
k3s upgrade homelab
Status: 🛠 Runbook | Last updated: 2026-05-08
Overview
Use this skill when updating k3s in this homelab. It covers: safety model, node access, upgrade commands, per-machine workflow, and recovery if an agent ends up on a control-plane node.
Safety model
- Update only one machine at a time.
- Do not move to the next machine until the current one is confirmed healthy and the cluster is stable.
- If a node fails to recover, stop and fix that node before touching any other machine.
Node access
- Load the
k8s-homelab skill for node access details and general homelab rules.
- Always verify the node-specific SSH user before running the upgrade command. Do not assume every node uses the same user.
Upgrade commands
Use the command matching the node role you are upgrading. Control-plane / server node (existing additional control-plane node):
k3sup join --server --ip <ip> --server-ip <healthy-server-ip> --user <ssh-user> --server-user <server-ssh-user> --ssh-key ~/.ssh/id_ed25519 --k3s-channel stable --no-extras
Worker / agent node:
k3sup join --ip <ip> --server-ip <healthy-server-ip> --user <ssh-user> --server-user <server-ssh-user> --ssh-key ~/.ssh/id_ed25519 --k3s-channel stable
Required process for each machine
- Confirm which machine is being updated and ensure no other node is mid-upgrade.
- Confirm whether the node is a control-plane node or a worker node.
- Confirm the correct SSH user for that specific node.
- Confirm
--server-ip points to another healthy control-plane node, never the node being upgraded.
- Before running
k3sup, inspect the active k3s unit on that machine.
- For control-plane nodes, inspect
/etc/systemd/system/k3s.service and record any arguments present in ExecStart.
- For worker nodes, verify the active unit is
k3s-agent.service.
- Run the role-correct
k3sup command for that machine.
- For control-plane nodes, re-check
/etc/systemd/system/k3s.service because k3sup may overwrite it.
- Restore any previously present
ExecStart arguments that were lost.
- Run
systemctl daemon-reload after editing the unit file.
- Ensure the correct service is running again after the unit is restored (
k3s on control-plane, k3s-agent on workers).
- Verify the node is updated and back to
Ready before continuing.
Verified service paths and ExecStart preservation
- Control-plane nodes use
/etc/systemd/system/k3s.service.
- Worker nodes may not have that path. Their active unit is
k3s-agent.service.
- When
k3sup rewrites the control-plane k3s.service unit, restore any custom arguments that existed before the update.
Recovery and verification
See references/recovery-and-verification.md for recovery if agent mode was used on a control-plane node, and the full verification checklist (including systemctl cat k3s and kubectl get nodes) before moving to the next machine.
Workflow expectations
- Keep the upgrade sequence serialized: one node only.
- Be explicit about which node is in progress and which are still pending.
- If unclear about restart timing after
daemon-reload, verify the unit, reload systemd, then confirm k3s health before proceeding.
See also
Routed by the coilyco-ops-investigation meta-skill alongside daily-operational as the live owner of k3s homelab cluster work. When a node, pod, or systemd-unit issue surfaces, this skill (upgrade work) or daily-operational (day-to-day pod health) is the destination, not the meta, which only ensures the universal first moves run first.
