| name | monitoring-ops |
| description | Monitoring operations skill. For VictoriaMetrics metric queries, Prometheus PromQL analysis, AlertManager alert management,
cluster health assessment, resource usage analysis, and performance bottleneck identification.
触发场景：指标查询、告警分析、集群巡检、资源使用率、性能问题、容量规划。
|
| allowed-tools | query_prometheus_metric get_alerts get_cluster_metrics get_node_info get_k8s_components_status |
Monitoring Operations Guide
Cluster Health Check Flow
1. Cluster Overview
# Node status
kube_node_status_condition{condition="Ready",status="true"}
# Pod status counts
count(kube_pod_status_phase{phase="Running"})
count(kube_pod_status_phase{phase="Pending"})
count(kube_pod_status_phase{phase="Failed"})
2. Resource Usage
# Node CPU usage
100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
# Node memory usage
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
# Pod CPU usage (based on Limit)
sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (pod, namespace)
/ sum(kube_pod_container_resource_limits{resource="cpu"}) by (pod, namespace) * 100
# Pod memory usage (based on Limit)
sum(container_memory_working_set_bytes{container!=""}) by (pod, namespace)
/ sum(kube_pod_container_resource_limits{resource="memory"}) by (pod, namespace) * 100
3. Key Metric Thresholds
| Metric | Warning | Critical | Description |
|---|
| CPU usage | >70% | >85% | For 5 minutes |
| Memory usage | >75% | >90% | For 5 minutes |
| Disk usage | >80% | >90% | Single check |
| Pod restarts | >3 | >10 | Last 1 hour |
| PVC usage | >80% | >90% | Single check |
4. API Server Health
# API Server request latency P99
histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket{verb!="WATCH"}[5m]))
# API Server error rate
sum(rate(apiserver_request_total{code=~"5.."}[5m])) / sum(rate(apiserver_request_total[5m])) * 100
5. ETCD Health
# ETCD Leader changes
changes(etcd_server_is_leader[1h])
# ETCD database size
etcd_mvcc_db_total_size_in_bytes
# ETCD disk fsync latency
histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m]))
Alert Analysis Flow
- Get current active alert list
- Group by severity (critical > warning > info)
- For each alert:
- Query related metric trends (last 1h/6h/24h)
- Check associated Pod/Node status
- Analyze alert trigger pattern (first vs frequent)
- Output alert summary and suggested actions
Capacity Planning
# CPU trend forecast (7-day linear regression)
predict_linear(node_cpu_seconds_total{mode="idle"}[7d], 30*24*3600)
# Disk usage trend forecast
predict_linear(node_filesystem_avail_bytes[7d], 30*24*3600)
