| name | deployment-operations |
| description | This skill should be used when standing up the monitoring stack against a real
or local cluster, or when diagnosing one that is not healthy — pods that will
not start, storage that will not attach, config changes that appear to do
nothing, or a teardown that hangs. Applies to Helm and Terraform installs and to
the kind E2E tiers.
|
Deployment Operations
Most of what belongs here is in the docs, because operators hit the same problems.
Go there first; this file exists to route you and to record the habits that keep a
diagnosis honest.
Read first
Name the cluster, every time
kubectl and helm both take the current kubeconfig context when you do not say
otherwise, and these operations install, restart, and delete. A context left
pointing at production is the failure mode.
- The
make e2e-* targets pin KIND_CONTEXT; the Terraform test substrate pins
its providers. Do not add a command that inherits the ambient context.
kubectl config current-context ignores --context — it reports the
configured context, not the one a command used. Do not use it to prove which
cluster you touched. Capture kubectl cluster-info instead; the server address
is unambiguous.
Before diagnosing anything
A timeout is not a duration problem. "helm upgrade timed out" almost never
means something was slow — assume something is broken, and raising the timeout is
almost always the wrong response. Identical whether you ran helm or Terraform
ran it for you. Inspect events first, then pod status, then the previous
container's logs. See
Troubleshooting.
Check your credentials before your work. AWS and GCP sessions commonly last
12 hours, so a morning login lapses mid-afternoon — and an expired session breaks
everything at once, including things you did not touch. When a failure looks
impossibly broad, that breadth is the tell. Re-authenticate, then debug.
Three habits that prevent wrong conclusions
Assert on recent data, not on any data. Loki's filesystem store survives a pod
restart and WAL-replayed streams count toward streams_created_total, so an
unbounded query and that counter both pass against a stack that stopped ingesting
an hour ago. Bound the window, and keep it meaningfully shorter than how long a
break would have lasted.
A green helm upgrade is not evidence the change took effect. Alloy's config
arrives partly through envFrom, and environment variables are fixed at container
start. Check the ConfigMap and whether the pod restarted.
Read the render output. The chart's validators print to NOTES.txt, and a
warning there is usually the earliest signal. On the Terraform path they surface
in the apply log — which is why the module disables subchart notes, so ours are
not buried.
When you cannot reproduce it later
The cluster dies with the CI runner, and a red E2E with no artifacts costs more
than the test saves. test/e2e/dump-diagnostics.sh
collects pods, events, storage, describes for unhealthy pods, all container logs
including --previous, the four Alloy ConfigMaps, and the cluster's identity.
Run it before deleting anything, and extend it rather than gathering by hand.
Things that are the cloud, not us
Worth recognising quickly, because they look like misconfiguration:
RESOURCE_POOL_EXHAUSTED / GCE out of resources — a capacity stockout.
Narrow machine families (Arm, bundled local SSD) hit it more often.pd-balanced disk type cannot be used by <machine-type> — GCP C4/N4 take
only Hyperdisk, and no default GKE StorageClass qualifies.- Workload identity cannot be tested on kind. There is no OIDC issuer an IAM
provider trusts, so IRSA, GKE Workload Identity, and Entra Workload ID are only
exercised against real clouds. The chart validators assert the shape of that
config instead, which is why a mismatched annotation fails at render time.
Storage changes are not free
volumeClaimTemplates are immutable, and Kubernetes does not garbage-collect
PVCs created from them — persistentVolumeClaimRetentionPolicy defaults to
Retain. So reinstalling does not reset a volume's class or size; the new
StatefulSet binds the old PVC. Delete PVCs deliberately, and know what is on them
before you do (see the per-workload notes in the troubleshooting page).
