| name | homelab-runbooks |
| description | Use when powering on a sleeping DR node, converging DNS records for a new ingress vhost, or bringing up OpenBao/Terrakube JWT identity — homelab operational runbooks for power management, DNS, and secrets-engine bring-up. |
Homelab Operations Runbooks
Generic, vendor/topology-neutral runbooks for recurring homelab operator
tasks. Every identifier below is a placeholder — substitute your own node
names, domain, and env-var values; never commit real ones.
Operating note: secrets are var-names, never values
Every credential in this skill is referenced by its environment/secret
variable name (e.g. IDRAC_USERNAME, <NODE>_BMC_HOSTNAME), never a
literal value. When writing runbooks, commits, or PRs in a public repo:
reference the variable name that holds a secret, not the secret itself, and
never a real hostname/IP/node name either — use <node> / proxmox-N /
<domain> placeholders. See also: always address a host by its FQDN,
never a hardcoded IP — IPs are derived from a VLAN + host-id scheme and
silently go stale the moment a guest moves.
Runbook 1: Power-managed DR node wake (Dell PowerEdge / iDRAC)
Some DR/standby nodes in a Dell PowerEdge estate sleep nightly to save power
and are woken on demand via their iDRAC BMC (ipmitool), same procedure for
any node in the class:
ipmitool -I lanplus -H <node>-bmc -U "$IDRAC_USERNAME" -E chassis power on
-E reads the password from the IPMITOOL_PASSWORD env var — never
put the password on the command line or in shell history.IPMITOOL_PASSWORD and IDRAC_USERNAME are sourced from the secrets
engine at runtime; the BMC hostname comes from a per-node var, generalized
here as <NODE>_BMC_HOSTNAME.- On the target host, the credential is delivered via a root-only
EnvironmentFile (mode 0600), never inlined in a unit file or script.
Automation pattern (e.g. an idrac_power Ansible role/play): gate the
power-on task on a per-node flag such as pve_power_managed: true, then
ipmitool ... chassis power on followed by wait_for_connection (poll SSH
until the node answers) before running any converge against it.
Gotcha — BMC network isolation. The BMC/OOB network is usually its own
isolated VLAN that the operator's own workstation cannot route to. Delegate
the ipmitool call to a controller host that already has OOB network
reachability (delegate_to: in Ansible) rather than trying to run it from
wherever the operator happens to be.
Runbook 2: DNS ingress-alias record convergence
Internal split-horizon DNS (e.g. an HA pair of authoritative resolvers)
creates the A/CNAME records that let a reverse-proxy ingress vhost resolve
inside the network. These records are generated from the IaC inventory's
list of ingress entries — they are not created by hand.
- Add or edit the ingress row in the inventory (new vhost, new backend
target).
- Re-run the DNS-tagged converge for the DNS role/play (e.g.
--tags dns or equivalent) so the new record set is applied to both
members of the HA pair.
- Verify resolution from inside the network:
dig <new-vhost>.<domain>.
Gotcha. A guest restart does not guarantee its DNS record survives —
re-run the DNS converge after any restart of a DNS-serving guest, and after
any ingress inventory change, before assuming a new vhost resolves.
Runbook 3: Secrets-engine (OpenBao) → workload-identity (Terrakube) bring-up order
A central secrets engine acts as the machine-identity authority. A JWT auth
method on it validates workload tokens against an external OIDC issuer
(e.g. a CI/CD identity provider). Bring-up must happen in this order, or the
issuer-config write fails:
- DNS record for the issuer exists — the secrets engine host must be
able to resolve the issuer's hostname.
- Issuer service is up — its
/.well-known/openid-configuration
discovery document must be reachable from the secrets-engine host.
- Secrets-engine issuer config is written — only after 1 and 2 hold;
otherwise the write fails with an "error checking oidc discovery URL"
error, which almost always means step 1 or 2 wasn't satisfied yet, not a
config-syntax problem.
- Workspace/role bindings are created — scoped roles that map validated
JWT claims to a policy, created last.
Gotcha — default-on secrets engines. Engines that are enabled by
default (e.g. a cloud-provider or SCM secrets engine) fail loud at converge
time if their backing credentials haven't been seeded yet. Skip them
explicitly until seeded, e.g. -e '{"<engine>_enabled": false}' (pass as a
JSON boolean, or ensure the consuming role coerces the value with | bool
in Jinja) rather than letting the play fail partway through.
Related
infra-standards (this marketplace) — VMID/IP ranges and the
Terraform-to-Ansible inventory contract.code-standards / secrets-policy org rules — the four-tier secrets
hierarchy this skill's "var-name, not value" note derives from.
