osdc-observability

OSDC observability: monitoring (metrics) and logging pipelines, three-Alloy architecture, Grafana Cloud Loki + Mimir queries, label strategy, credential setup, and troubleshooting. Applies to ~/meta/ci-infra/osdc. Load when working on monitoring, logging, Alloy, or querying logs.

仓库: pytorch/ci-infra
最近来源活动: 2026年7月21日 18:16
检测到的 SKILL.md 语言: 英语
星标: 17
分支: 14

安装方式

默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。

检查来源文件

决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。

返回搜索返回安装

name	osdc-observability
description	OSDC observability: monitoring (metrics) and logging pipelines, three-Alloy architecture, Grafana Cloud Loki + Mimir queries, label strategy, credential setup, and troubleshooting. Applies to ~/meta/ci-infra/osdc. Load when working on monitoring, logging, Alloy, or querying logs.

OSDC Observability: Monitoring + Logging

OSDC has two observability pipelines, both pushing to Grafana Cloud. They use three Grafana Alloy installations to avoid RBAC and config collisions:

Pipeline	Component	Location	Alloy Mode	Namespace	Helm Release
Metrics	`modules/monitoring/`	Module (opt-in)	Deployment (2 replicas, clustered)	`monitoring`	`alloy`
Logs	`modules/logging/`	Module (opt-in)	DaemonSet (one per node)	`logging`	`alloy-logging`
Events	`modules/logging/`	Module (opt-in)	Deployment (1 replica)	`logging`	`alloy-events`

Both modules are secret-gated: Alloy only installs if a grafana-cloud-credentials secret exists in the respective namespace. The secrets have different keys (metrics uses username/password; logging uses loki-username/loki-api-key-write/). All Grafana Cloud URLs come from , not the secret.

loki-api-key-read

clusters.yaml

Type	Name	Target Namespace	What it monitors
ServiceMonitor	apiserver	default	Kubernetes API server metrics
ServiceMonitor	arc-controller	arc-systems	ARC controller metrics
ServiceMonitor	buildkit	buildkit	BuildKit builder pod metrics
ServiceMonitor	buildkit-haproxy	buildkit	BuildKit HAProxy load balancer metrics
ServiceMonitor	dcgm-exporter	monitoring	NVIDIA GPU metrics (DCGM) — file lives directly under `monitors/dcgm-servicemonitor.yaml`, not in the `servicemonitors/` subdir
ServiceMonitor	harbor	harbor-system	Harbor exporter metrics
ServiceMonitor	karpenter	karpenter	Karpenter controller metrics
ServiceMonitor	keda	keda	KEDA operator metrics (`keda_` keep-list; `_bucket` dropped)
ServiceMonitor	node-compactor	kube-system	Node compactor metrics
ServiceMonitor	pushgateway	monitoring	Prometheus Pushgateway metrics
ServiceMonitor	pypi-cache	pypi-cache	PyPI cache nginx + pypiserver metrics
PodMonitor	arc-listeners	arc-systems	ARC listener pods metrics (interval `3m` — the only sub-60s exception; all other monitors run at 60s to halve Grafana DPM billing)
PodMonitor	coredns	kube-system	CoreDNS resolver metrics
PodMonitor	nodelocaldns	kube-system	NodeLocal DNSCache — TWO endpoints per pod: `:9253` (`coredns_` plugin metrics from the Corefile `prometheus` directive) AND `:9353` (`coredns_nodecache_` setup-error counters emitted by the binary itself, NOT a CoreDNS plugin). Adding only `:9253` would silently miss `coredns_nodecache_setup_errors_total` — the alert metric.

KSM allowlist (modules/monitoring/helm/values.yaml, kube-state-metrics extraArgs): --metric-allowlist=kube_(daemonset|deployment|pod|namespace|node|statefulset|persistentvolume|horizontalpodautoscaler|job)_.+ — drops entire metric families at the exporter before they're scraped. KSM's monitor.metricRelabelings then drops kube_node_status_addresses, benign Completed terminations (kube_pod_container_status_(last_)?terminated_reason;Completed), exit-code-0 terminations (kube_pod_container_status_last_terminated_exitcode;0), and routine status reasons (kube_pod_status_reason;(Shutdown|NodeAffinity)) — keeping only real errors like Evicted, NodeLost, UnexpectedAdmissionError.
ServiceMonitor metricRelabelings (modules/monitoring/kubernetes/monitors/servicemonitors/*.yaml): per-target keep/drop rules. Examples: apiserver keeps only apiserver_request_total|apiserver_request_terminations_total; karpenter keeps a focused list and drops histograms; harbor/buildkit drop go_.*|process_.*|promhttp_.*. CoreDNS/API server low-value metrics are dropped here. node-exporter uses a tight keep-allowlist: only node_memory_MemAvailable_bytes, node_memory_MemTotal_bytes, node_nf_conntrack_entries, node_nf_conntrack_entries_limit survive — so CPU/disk/network device metrics are NOT in Mimir by design. (cAdvisor relabel rules would also live here — but the kubelet ServiceMonitor is currently off under IPv6-only EKS, so no cAdvisor metrics reach Alloy.)
Alloy cost_control relabel (modules/monitoring/helm/alloy-values.yaml, prometheus.relabel "cost_control"): final pass before remote_write. Five rule groups:
- KSM low-value: kube_.*_created, kube_.*_metadata_resource_version, kube_secret_.*, kube_configmap_.*, kube_endpoint_.*, kube_lease_.* — blanket drop
- Control-plane-only scoping: kube_pod_container_status_restarts_total, container_memory_working_set_bytes, container_memory_rss — kept ONLY for arc-systems|karpenter|harbor-system|monitoring|logging|buildkit. Two-step replace+drop pattern (RE2 has no lookahead). Note: the two container_memory_* metrics are sourced from cAdvisor via the kubelet ServiceMonitor — currently disabled under IPv6-only EKS — so this scoping rule is effectively a no-op for those two metrics until a custom IPv6-aware kubelet ServiceMonitor lands
- Misc high-cardinality: kubernetes_feature_enabled — blanket drop
- ARC histogram buckets: gha_job_(execution|startup)_duration_seconds_bucket — dropped (sum/count kept)
- Runtime internals: go_.*, process_.*, promhttp_.*, prometheus_operator_.* — blanket drop

defaults:
  monitoring:
    grafana_cloud_url: "https://prometheus-prod-36-prod-us-west-0.grafana.net/api/prom/push"
    grafana_cloud_read_url: "https://prometheus-prod-36-prod-us-west-0.grafana.net/api/prom"
  alloy_chart_version: "1.6.2"

logging:
  grafana_cloud_loki_url: "https://logs-prod-021.grafana.net/loki/api/v1/push"

Label	Type	Query Usage
`cluster`	external_label	`{cluster="..."}`
`unit`	indexed	`{unit="..."}` (journal logs)
`level`	indexed	`{level="error"}` (journal, from priority mapping)
`node`	structured metadata (Loki 3.x)	`{...} \| node="..."` (pipe, NOT inside `{}`)
`type`, `kind`	indexed (events only)	`{type="Warning", kind="Pod"}`
`reason`, `sourcecomponent`	structured metadata (events only)	`{kind="Pod"} \| reason="..."`

kubectl create secret generic grafana-cloud-credentials \
  -n monitoring \
  --from-literal=username='<GRAFANA_CLOUD_METRICS_USER_ID>' \
  --from-literal=password='<API_KEY_WITH_METRICS_WRITE_SCOPE>'

# Separate read secret for Mimir queries (see "Querying" below)
kubectl create secret generic grafana-cloud-read-credentials \
  -n monitoring \
  --from-literal=username='<GRAFANA_CLOUD_METRICS_USER_ID>' \
  --from-literal=password='<API_KEY_WITH_METRICS_READ_SCOPE>'

kubectl create namespace logging
kubectl create secret generic grafana-cloud-credentials \
  -n logging \
  --from-literal=loki-username='<GRAFANA_CLOUD_LOKI_USER_ID>' \
  --from-literal=loki-api-key-write='<API_KEY_WITH_LOGS_WRITE_SCOPE>' \
  --from-literal=loki-api-key-read='<API_KEY_WITH_LOGS_READ_SCOPE>'

LOKI_USER=$(NO_PROXY="$NO_PROXY,.eks.amazonaws.com" no_proxy="$no_proxy,.eks.amazonaws.com" \
  mise exec -- kubectl get secret grafana-cloud-credentials -n logging \
  -o jsonpath='{.data.loki-username}' | base64 -d)

LOKI_READ_KEY=$(NO_PROXY="$NO_PROXY,.eks.amazonaws.com" no_proxy="$no_proxy,.eks.amazonaws.com" \
  mise exec -- kubectl get secret grafana-cloud-credentials -n logging \
  -o jsonpath='{.data.loki-api-key-read}' | base64 -d)

LOKI_URL="https://logs-prod-021.grafana.net"

# Journal logs by unit (kubelet, containerd)
curl -s -u "$LOKI_USER:$LOKI_READ_KEY" \
    "$LOKI_URL/loki/api/v1/query_range" \
    --data-urlencode 'query={cluster="<cluster-name>", unit="kubelet.service"}' \
    --data-urlencode "limit=100" \
    --data-urlencode "start=$(date -u -v-1H +%s)000000000" \
    --data-urlencode "end=$(date -u +%s)000000000" | jq .

# Filter by node (structured metadata — pipe syntax)
curl -s -u "$LOKI_USER:$LOKI_READ_KEY" \
    "$LOKI_URL/loki/api/v1/query_range" \
    --data-urlencode 'query={cluster="<cluster-name>", unit="kubelet.service"} | node="<node-name>"' \
    --data-urlencode "limit=100" \
    --data-urlencode "start=$(date -u -v-1H +%s)000000000" \
    --data-urlencode "end=$(date -u +%s)000000000" | jq .

# Pod logs by namespace + container
curl -s -u "$LOKI_USER:$LOKI_READ_KEY" \
    "$LOKI_URL/loki/api/v1/query_range" \
    --data-urlencode 'query={cluster="<cluster-name>", namespace="arc-runners", container="runner"}' \
    --data-urlencode "limit=100" \
    --data-urlencode "start=$(date -u -v-1H +%s)000000000" \
    --data-urlencode "end=$(date -u +%s)000000000" | jq .

# Filter pod logs by specific pod (structured metadata)
curl -s -u "$LOKI_USER:$LOKI_READ_KEY" \
    "$LOKI_URL/loki/api/v1/query_range" \
    --data-urlencode 'query={cluster="<cluster-name>", namespace="arc-runners"} | pod="<pod-name>"' \
    --data-urlencode "limit=100" \
    --data-urlencode "start=$(date -u -v-6H +%s)000000000" \
    --data-urlencode "end=$(date -u +%s)000000000" | jq .

# List labels / values
curl -s -u "$LOKI_USER:$LOKI_READ_KEY" "$LOKI_URL/loki/api/v1/labels" | jq .
curl -s -u "$LOKI_USER:$LOKI_READ_KEY" "$LOKI_URL/loki/api/v1/label/cluster/values" | jq .

MIMIR_USER=$(NO_PROXY="$NO_PROXY,.eks.amazonaws.com" no_proxy="$no_proxy,.eks.amazonaws.com" \
  mise exec -- kubectl get secret grafana-cloud-read-credentials -n monitoring \
  -o jsonpath='{.data.username}' | base64 -d)

MIMIR_PASS=$(NO_PROXY="$NO_PROXY,.eks.amazonaws.com" no_proxy="$no_proxy,.eks.amazonaws.com" \
  mise exec -- kubectl get secret grafana-cloud-read-credentials -n monitoring \
  -o jsonpath='{.data.password}' | base64 -d)

MIMIR_URL="https://prometheus-prod-36-prod-us-west-0.grafana.net/api/prom"

# Instant query
curl -s -u "$MIMIR_USER:$MIMIR_PASS" \
    "$MIMIR_URL/api/v1/query" \
    --data-urlencode 'query=up{cluster="<cluster-name>"}' | jq .

# Range query
curl -s -u "$MIMIR_USER:$MIMIR_PASS" \
    "$MIMIR_URL/api/v1/query_range" \
    --data-urlencode 'query=rate(node_cpu_seconds_total{cluster="<cluster-name>", mode="idle"}[5m])' \
    --data-urlencode "start=$(date -u -v-1H +%s)" \
    --data-urlencode "end=$(date -u +%s)" \
    --data-urlencode "step=60" | jq .

Alloy not installed: check kubectl get secret grafana-cloud-credentials -n {logging,monitoring} — deploy is skipped without it
Alloy memory: all three Alloys have explicit Go runtime tuning via deploy.sh extraEnv. Centralized Deployments get generous headroom because their cost multiplies by 1 (not by node count); the DaemonSet is tuned tighter because its cost multiplies by node count.
- Monitoring Alloy (Deployment, 2 replicas): GOMEMLIMIT=7000MiB (~85% of 8Gi cgroup), no GOGC override.
- Logging DaemonSet (alloy-logging): requests=1Gi/limits=2Gi, GOGC=200, GOMEMLIMIT=1800MiB (~85% of 2Gi cgroup) for high-throughput CI nodes.
- Events Deployment (alloy-events): GOGC=200, GOMEMLIMIT=3500MiB (~85% of 4Gi cgroup). Default upstream is 1Gi — bump only if kubectl describe pod shows OOMKilled. When raising the cgroup limit, also bump GOMEMLIMIT to ~85% of the new ceiling.
DCGM exporter OOM: pod is sized requests=256Mi/limits=512Mi with GOMEMLIMIT=410MiB and 60s collection interval (--collect-interval=60000). Aggressive limits were chosen because the per-GPU metric set was trimmed to 23 metrics; if DCGM is OOMKilled after re-introducing metrics, bump resources.limits.memory AND GOMEMLIMIT together (keep GOMEMLIMIT at ~80% of the new cgroup ceiling).
Alloy did not pick up secret rotation: monitoring deploy.sh runs kubectl rollout restart deployment/alloy after every successful Helm upgrade and waits on rollout status. Secret values are referenced via valueFrom.secretKeyRef, so Kubernetes will NOT restart pods when the secret value changes — only the explicit rollout does. If you rotate credentials without redeploying, force a rollout manually.
Missing pod logs: container stdout/stderr is intentionally NOT shipped. Look at GitHub Actions workflow logs (for runner jobs), kubectl logs (while the pod is still alive), or pod events via the alloy-events Deployment in Loki ({cluster="X", kind="Pod"})
Sampling-related "missing" logs: not applicable — namespace-based sampling stages were removed with the pod-log pipeline
Rate limiting: not applicable — the per-pod rate-limit stage was removed with the pod-log pipeline. Journal logs go through loki.process "system_logs" which has no rate limit; volume is bounded by the per-unit keep filter on the journal source (kubelet|containerd|kernel|nvidia-*)
Journal path empty: EKS AL2023 uses /var/log/journal — hostPath uses DirectoryOrCreate so it won't crash, but no journal logs will appear if path is wrong
Dashboard publish failed: grafana-publish.yml runs mise run push --folder fvnzj9 from the grafana/ directory. Validation is gated by gcx resources validate; if it fails, the dashboard JSON in grafana/*.json is malformed or references a missing datasource. Reproduce locally with cd grafana && mise run validate --folder fvnzj9.