kubernetes-operations

name	kubernetes-operations
description	Kubernetes cluster operations on minikube including observability (Grafana, Prometheus, Alertmanager, Loki, Tempo), debugging (kubectl debug, ephemeral containers), and cluster management (ArgoCD). Use when working with cluster/manifests/, Kubernetes workloads, pods, deployments, operators, controllers, or cluster components.
context	fork
keywords	kubernetes, k8s, minikube, grafana, prometheus, loki, tempo, argocd, pod, クラスタ, 監視, kubectl, deployment

• Access Grafana at https://grafana.minikube.127.0.0.1.nip.io
• Disable ArgoCD selfHeal before manual changes, re-enable after
• Prometheus runs in Agent mode (remote-write only, no query engine) — Mimir is the only PromQL backend

Debugging

Investigating	Tool	Entry Point
Traces, metrics, logs, profiles	Grafana	https://grafana.minikube.127.0.0.1.nip.io
Pod startup failures, events	kubectl	kubectl get events -n <namespace>
In-container process/network	kubectl debug	Ephemeral container
kubectl connection refused despite VM running (clusters/users empty in ~/.kube/config)	minikube	minikube update-context
ArgoCD Application stuck OutOfSync/Missing	kubectl	kubectl get application -n argocd <name> -o jsonpath='{.status.operationState.message}'

Observability Investigation (via Grafana)

Use Grafana for all observability signals. Do NOT query backends (Tempo, Mimir, Loki) directly.

1. Get query parameters - Check cluster/manifests/<app>/ for namespace, labels, OTEL_SERVICE_NAME (see Queries for parameter locations)
2. Open Grafana - Access https://grafana.minikube.127.0.0.1.nip.io in browser
3. Select datasource and query - Use appropriate signal:

Signal	Backend	Datasource	Query Language	Use Case
Traces	Tempo	Tempo	TraceQL	Request flow, latency, access destinations
Metrics	Mimir (Prometheus)	Mimir	PromQL	Resource usage, HTTP rates, alerting
Logs	Loki	Loki	LogQL	Error investigation, audit
Profiles	Pyroscope	Pyroscope	Flamegraph UI	CPU/memory hotspots
Probes	Blackbox Exporter	Mimir	PromQL	Endpoint reachability

Symptom	Signal	Query Approach
Errors in logs	Loki → Tempo	Extract traceid from logs, trace in Tempo
Latency/5xx	Tempo	Search traces with status = error
Unknown outbound dependencies	Tempo	Search traces, inspect spans for outbound calls
Resource saturation	Mimir	Query CPU/memory metrics
High CPU/memory	Pyroscope	Check flamegraphs

Pod Direct Investigation (via kubectl)

Use kubectl when Grafana cannot answer the question (e.g., pod not starting, container-level inspection).

Symptom	Action
Pod not starting	kubectl get events -n <namespace>
CrashLoopBackOff	kubectl logs <pod> -n <namespace> --previous
Network connectivity	kubectl debug with ephemeral container
Process inspection	kubectl debug with ephemeral container
JVM effective flags	kubectl exec → jcmd -l, then jcmd <pid> VM.flags (see below)

JVM Flag Inspection

Use kubectl exec -c <container>, not kubectl debug — the ephemeral-container image ships no JDK. PID 1 is often tini, so jcmd 1 fails to attach; find the real PID first. jcmd -l usually lists it, but returns nothing on images without hsperfdata, so fall back to scanning cmdlines:

for p in /proc/[0-9]*; do tr '\0' ' ' < $p/cmdline | grep -q '^java ' && echo ${p#/proc/}; done

Never run java -XX:+PrintFlagsFinal -version, which starts a second JVM in the same cgroup and can OOM-kill the process being inspected.

Ephemeral Container

kubectl debug <pod-name> -n <namespace> \
  --profile=restricted \
  --image=ghcr.io/hippocampus-dev/hippocampus/ephemeral-container:main \
  --target=<container-name> \
  -- <command>

Note: Do not use -it flag when executing commands. It causes output streaming issues.

Manual Changes

Required when directly modifying ArgoCD-managed resources (e.g., kubectl apply, kubectl patch, kubectl delete) outside of the GitOps workflow.

Operation	Requires Manual Changes
kubectl apply/patch/delete on an ArgoCD-managed resource	Yes
kubectl delete pod where a ReplicaSet/StatefulSet/DaemonSet recreates it	No (ArgoCD manages the controller, not the pod)
Debugging (read-only: logs, events, debug)	No
Grafana queries	No
Editing manifests in repo	No (ArgoCD syncs automatically)

ArgoCD selfHeal Control

ArgoCD reverts manual changes unless selfHeal is disabled first. Always re-enable after — "after" means once the change is durably reflected in the manifest ArgoCD syncs from (committed and pushed, or already matching git), not merely once the live cluster looks healthy. If the backing manifest edit cannot yet be committed (e.g. a read-only .git), leave selfHeal disabled until it can be; re-enabling first reverts the live fix back to the stale manifest.

# Disable selfHeal
kubectl patch application <app-name> -n argocd --type=merge \
  -p '{"spec":{"syncPolicy":{"automated":{"selfHeal":false}}}}'

# Re-enable selfHeal (after work is complete)
kubectl patch application <app-name> -n argocd --type=merge \
  -p '{"spec":{"syncPolicy":{"automated":{"selfHeal":true}}}}'

selfHeal lives under automated; patching it at the syncPolicy root is silently pruned and leaves selfHeal enabled.

Re-enabling selfHeal restores the manifest ArgoCD manages, not what an operator derives from it. After reverting an experiment on a CR, check the operator-managed Deployment/StatefulSet directly — operators that merge resource fields rather than replacing them leave removed keys behind.

External Service Access

Services exposed via Istio Gateway follow a tiered host naming convention:

Host	Authentication	Access
{service}.minikube.127.0.0.1.nip.io	None	Local development
{service}.kaidotio.dev	OAuth2 (ext-authz)	Browser access
{service}-public.kaidotio.dev	Custom (JWT, method restriction, etc.)	Programmatic access excluded from OAuth2

The -public variant is used when OAuth2 is technically impossible (e.g., W3C Reporting API browser-generated requests) or when the service implements its own AuthorizationPolicy (e.g., GitHub Actions OIDC token validation).

To access a service's data locally, use https://{service}.minikube.127.0.0.1.nip.io.

Observability Stack Manifests

Component	Path
Grafana	cluster/manifests/grafana/
Tempo	cluster/manifests/tempo/
Mimir	cluster/manifests/mimir/
Loki	cluster/manifests/loki/
Pyroscope	cluster/manifests/pyroscope/
Prometheus	cluster/manifests/prometheus/
Fluentd	cluster/manifests/fluentd/
OpenTelemetry	cluster/manifests/otel-agent/, cluster/manifests/otel-collector/

Reference

If writing observability queries: See Queries

If an ArgoCD Application failed to sync and never retried: See ## Sync Wave in .claude/rules/cluster/manifests/argocd-applications.md