| name | monitoring-ops |
| description | Operate Prometheus-compatible monitoring with kubectl, curl, PromQL, and VictoriaMetrics CLI/API endpoints.
Use for metrics, active alerts, cluster health checks, resource utilization, performance bottlenecks, SLOs, and capacity planning.
适用于指标查询、告警分析、集群巡检、资源使用率、性能问题和容量规划。
|
Monitoring Operations Guide
Cluster Health Check Flow
Confirm the kubectl context, discover the in-cluster or configured query endpoint, then use curl --get --data-urlencode 'query=<promql>' <endpoint>/api/v1/query. Prefer bounded time ranges and record the query and timestamp with each conclusion.
1. Cluster Overview
# Node status
kube_node_status_condition{condition="Ready",status="true"}
# Pod status counts
count(kube_pod_status_phase{phase="Running"})
count(kube_pod_status_phase{phase="Pending"})
count(kube_pod_status_phase{phase="Failed"})
2. Resource Usage
# Node CPU usage
100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
# Node memory usage
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
# Pod CPU usage (based on Limit)
sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (pod, namespace)
/ sum(kube_pod_container_resource_limits{resource="cpu"}) by (pod, namespace) * 100
# Pod memory usage (based on Limit)
sum(container_memory_working_set_bytes{container!=""}) by (pod, namespace)
/ sum(kube_pod_container_resource_limits{resource="memory"}) by (pod, namespace) * 100
3. Key Metric Thresholds
| Metric | Warning | Critical | Description |
|---|
| CPU usage | >70% | >85% | For 5 minutes |
| Memory usage | >75% | >90% | For 5 minutes |
| Disk usage | >80% | >90% | Single check |
| Pod restarts | >3 | >10 | Last 1 hour |
| PVC usage | >80% | >90% | Single check |
4. API Server Health
# API Server request latency P99
histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket{verb!="WATCH"}[5m]))
# API Server error rate
sum(rate(apiserver_request_total{code=~"5.."}[5m])) / sum(rate(apiserver_request_total[5m])) * 100
5. ETCD Health
# ETCD Leader changes
changes(etcd_server_is_leader[1h])
# ETCD database size
etcd_mvcc_db_total_size_in_bytes
# ETCD disk fsync latency
histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m]))
Alert Analysis Flow
- Get current active alert list
- Group by severity (critical > warning > info)
- For each alert:
- Query related metric trends (last 1h/6h/24h)
- Check associated Pod/Node status
- Analyze alert trigger pattern (first vs frequent)
- Output alert summary and suggested actions
Capacity Planning
# CPU trend forecast (7-day linear regression)
predict_linear(node_cpu_seconds_total{mode="idle"}[7d], 30*24*3600)
# Disk usage trend forecast
predict_linear(node_filesystem_avail_bytes[7d], 30*24*3600)
Keep alert-rule changes in Git, validate them with promtool check rules, show the diff, and require approval before apply or reload.
