running-stabilization-loop

name	running-stabilization-loop
description	Use after major refactors or risky infrastructure changes to run iterative stabilization loops in kube-test: clean state, run the v3 matrix (k3s/rke2, autoscaler, leapmicro/microos, multi-arch), proactively debug failures over SSH/journalctl, apply fixes, retest until green, then validate upgrade paths from the latest tag to the current release-candidate branch.

Running Stabilization Loop

Overview

Use this workflow when the goal is not only to detect failures, but to fix them and prove full regression safety after major changes.

Usage

/running-stabilization-loop

Buddy Prompt Template

Use this exact prompt when delegating the loop to another agent:

Run the full kube-hetzner v3 stabilization loop end-to-end, do not stop at pass/fail reporting.

Goals:
1) Make every matrix scenario pass from simple to complex.
2) Fix root causes in kube-hetzner (not workarounds in test harness) unless explicitly required.
3) Re-run full matrix after fixes to prove no regressions.
4) Validate upgrade path from latest git tag to the current release-candidate branch for both k3s and rke2.
5) When Tailscale credentials are available, include the Tailscale node-transport
   scenarios: secure single-network Tailnet access, plus Flannel with multiple
   Hetzner Networks, route auto-approval, autoscaler scale-up/down,
   `tailscale status`, and cross-network pod traffic.
6) Include final v3 polish scenarios when touched: Cilium Gateway API with
   `enable_kube_proxy=false`, embedded registry mirror on k3s or RKE2, endpoint
   output sanity, and example validation.

Hard requirements:
- Use /Users/karim/.ssh/id_ed25519 with IdentitiesOnly=yes for all SSH.
- Use /Volumes/MysticalTech/Code/kube-test/run_v3_matrix.sh and summary-*.md as source of truth.
- If RKE2/K3s apply hangs, SSH during apply and inspect journalctl/cloud-init immediately.
- If destroy hangs, run cleanup and delete stuck autoscaled servers via hcloud.
- Keep changes minimal and production-safe; run Terraform and OpenTofu validation before each rerun.
- For Tailscale node transport, verify single-network clusters work without
  route approval when route advertisement is disabled; for multinetwork, verify
  node-private routes are approved and SNAT is disabled for advertised routes.
  Do not mistake external overlay SSH access for supported Kubernetes node
  transport.
- For Cilium Gateway API, verify GatewayClass exists, Gateway is accepted, and
  HTTPRoute serves through the Hetzner LoadBalancer. For embedded registry
  mirror, verify `registries.yaml`, `embedded-registry: true`, peer ports, and
  image-pull behavior.

Deliverables:
- Final scenario matrix results with log paths.
- List of fixes made with files changed.
- Upgrade test result (tag -> release-candidate branch) and any required migration notes.
- Confirmation cleanup is complete and no khv3-* resources are left.