菜单
Production-grade observability stack with Prometheus metrics, Grafana dashboards, PromQL query language, alerting rules, and AI-powered anomaly detection for modern cloud-native applications
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
基于 SOC 职业分类
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/bobmatnyc/mcp-skillset --skill observability-with-prometheus-grafana命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
正在显示 SKILL.md
Build ultra-low-latency edge computing applications with Cloudflare Workers, Workers AI for LLM inference, V8 isolates, Durable Objects, and serverless patterns deployed across 330+ data centers worldwide
Production-grade FastAPI development with async patterns, Pydantic v2, dependency injection, ML/AI endpoint design, and modern Python best practices for building high-performance REST APIs
Production-grade PostgreSQL query optimization, indexing strategies, performance tuning, and modern features including pgvector for AI/ML workloads. Master EXPLAIN plans, query analysis, and database design for high-performance applications
| name | Observability with Prometheus & Grafana |
| skill_id | observability-monitoring |
| version | 1.0.0 |
| description | Production-grade observability stack with Prometheus metrics, Grafana dashboards, PromQL query language, alerting rules, and AI-powered anomaly detection for modern cloud-native applications |
| category | DevOps & Infrastructure |
| tags | ["observability","monitoring","prometheus","grafana","metrics","alerting","promql","sre","cloud-native","ai-observability"] |
| author | mcp-skillset |
| license | MIT |
| created | 2025-11-25T00:00:00.000Z |
| last_updated | 2025-11-25T00:00:00.000Z |
| toolchain | ["Prometheus 2.45+","Grafana 10.0+","Alertmanager","Node Exporter","Blackbox Exporter"] |
| frameworks | ["Prometheus","Grafana","OpenTelemetry"] |
| related_skills | ["terraform-infrastructure","fastapi-web-development","postgresql-optimization","systematic-debugging"] |
Master production observability with Prometheus and Grafana - the industry-standard monitoring stack for cloud-native applications. Learn metrics collection, PromQL query language, dashboard design, alerting, and AI-powered anomaly detection (Grafana AI Observability 2024).
# Always monitor these four metrics for every service:
# 1. Latency - How long requests take
http_request_duration_seconds_bucket{le="0.1", job="api"} 8500
http_request_duration_seconds_bucket{le="0.5", job="api"} 9800
http_request_duration_seconds_sum{job="api"} 2450
http_request_duration_seconds_count{job="api"} 10000
# 2. Traffic - How many requests
http_requests_total{method="GET", status="200"} 50000
# 3. Errors - How many requests fail
http_requests_total{method="POST", status="500"} 150
# 4. Saturation - How "full" is the service
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.2
from prometheus_client import Counter, Gauge, Histogram, Summary, Info
request_count = Counter(
,
,
[, , ]
)
request_count.labels(method=, endpoint=, status=).inc()
active_connections = Gauge(
,
)
active_connections.()
active_connections.inc()
active_connections.dec()
request_duration = Histogram(
,
,
buckets=[, , , , , ]
)
request_duration.time():
process_request()
response_size = Summary(
,
)
response_size.observe()
app_info = Info(, )
app_info.info({: , : })
When to use each type:
# Basic queries
http_requests_total # Instant vector (latest value)
http_requests_total[5m] # Range vector (5 minute window)
# Label matching
http_requests_total{method="GET"} # Exact match
http_requests_total{method=~"GET|POST"} # Regex match
http_requests_total{method!="OPTIONS"} # Not equal
# Aggregation operators
sum(rate(http_requests_total[5m])) by (status) # Requests per second by status
avg(http_request_duration_seconds) by (endpoint) # Average latency by endpoint
max(node_memory_MemTotal_bytes) without (instance) # Max memory across instances
# Rate and increase (for counters)
rate(http_requests_total[5m]) # Per-second rate over 5 minutes
increase(http_requests_total[1h]) # Total increase over 1 hour
irate(http_requests_total[5m]) # Instant rate (sensitive to spikes)
# Histogram quantiles (percentiles)
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) # p95 latency
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) # p99 latency
# Math and comparisons
(sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))) * 100 # Error rate percentage
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1 # Memory <10%
# Logical operators
(rate(http_requests_total[5m]) > 100) and (http_request_duration_seconds > 1)
# /etc/prometheus/alerts.yml
groups:
- name: api_alerts
interval: 30s
rules:
# High error rate
- alert: HighErrorRate
expr: |
(sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/ sum(rate(http_requests_total[5m])) by (service)) > 0.05
for: 5m
labels:
severity: critical
team: backend
annotations:
summary: "High error rate on {{ $labels.service }}"
description: "Error rate is {{ $value | humanizePercentage }} (threshold: 5%)"
runbook: "https://wiki.company.com/runbooks/high-error-rate"
# High latency (p99)
- alert: HighLatency
expr: |
histogram_quantile(0.99,
rate(http_request_duration_seconds_bucket[5m])
) > 1.0
for: 10m
labels:
severity: warning
team: backend
annotations:
summary: "P99 latency above 1s"
description: "P99 latency is {{ $value }}s on {{ $labels.instance }}"
# Service down
- alert: ServiceDown
expr: up{job="api"} == 0
for: 1m
labels:
severity: critical
page: true
annotations:
summary: "Service {{ $labels.instance }} is down"
# Database connection exhaustion
- alert: DatabaseConnectionsHigh
expr: |
pg_stat_database_numbackends / pg_settings_max_connections > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "Database connections at {{ $value | humanizePercentage }}"
# Alertmanager configuration (/etc/alertmanager/alertmanager.yml)
route:
receiver: 'team-backend'
group_by: ['alertname', 'cluster']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- match:
severity: critical
receiver: 'pagerduty-critical'
- match:
severity: warning
receiver: 'slack-warnings'
receivers:
- name: 'pagerduty-critical'
pagerduty_configs:
- service_key: '<PAGERDUTY_KEY>'
- name: 'slack-warnings'
slack_configs:
- api_url: 'https://hooks.slack.com/services/XXX'
channel: '#alerts'
text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
{
"dashboard": {
"title": "API Performance Dashboard",
"panels": [
{
"title": "Request Rate (RPS)",
"targets": [{
"expr": "sum(rate(http_requests_total[5m])) by (service)",
"legendFormat": "{{ service }}"
}],
"type": "graph"
},
{
"title": "Error Rate (%)",
"targets": [{
"expr": "(sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m]))) * 100",
"legendFormat": "Error Rate"
}],
"type": "graph",
"alert": {
"conditions": [{"evaluator": {"params": [5], "type": "gt"}}],
"frequency": "60s",
"message": "Error rate above 5%"
}
},
{
"title": "Latency Percentiles",
"targets": [
{
"expr": "histogram_quantile(0.50, rate(http_request_duration_seconds_bucket[5m]))",
"legendFormat": "p50"
},
{
"expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))",
"legendFormat": "p95"
},
{
"expr": "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))",
"legendFormat": "p99"
}
],
"type": "graph"
}
]
}
}
# Format: <namespace>_<subsystem>_<name>_<unit>
http_requests_total # Counter: total requests
http_request_duration_seconds # Histogram: request duration
process_cpu_seconds_total # Counter: CPU time
node_memory_bytes # Gauge: memory in bytes
# Use base units:
- seconds (not milliseconds)
- bytes (not KB/MB)
- ratio (0.0-1.0, not percentage)
# GOOD: Low-cardinality labels
http_requests_total{method="GET", status="200", endpoint="/api/users"}
# BAD: High-cardinality labels (creates too many time series)
http_requests_total{user_id="12345"} # DON'T: user_id has millions of values!
http_requests_total{ip_address="192.168.1.1"} # DON'T: IP has too many values
# GOOD: Aggregate high-cardinality data
user_requests_total # Single metric without user_id label
Rule of thumb: Total cardinality = product of label values. Keep <10,000 per metric.
# /etc/prometheus/rules.yml
groups:
- name: api_performance
interval: 30s
rules:
# Pre-calculate p99 latency (expensive query)
- record: job:http_request_duration_seconds:p99
expr: |
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le)
)
# Pre-calculate error rate
- record: job:http_requests:error_rate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
/ sum(rate(http_requests_total[5m])) by (job)
# Use in dashboards/alerts:
# job:http_request_duration_seconds:p99 > 1.0
from fastapi import FastAPI, Request
from prometheus_client import Counter, Histogram, Gauge, make_asgi_app
import time
app = FastAPI()
# Metrics
REQUEST_COUNT = Counter(
'http_requests_total',
'Total HTTP requests',
['method', 'endpoint', 'status']
)
REQUEST_DURATION = Histogram(
'http_request_duration_seconds',
'HTTP request duration',
['method', 'endpoint']
)
ACTIVE_REQUESTS = Gauge(
'http_requests_in_progress',
'Number of HTTP requests in progress',
['method', 'endpoint']
)
@app.middleware("http")
async def prometheus_middleware(request: Request, call_next):
method = request.method
endpoint = request.url.path
ACTIVE_REQUESTS.labels(method=method, endpoint=endpoint).inc()
start_time = time.time()
response = await call_next(request)
duration = time.time() - start_time
REQUEST_COUNT.labels(
method=method,
endpoint=endpoint,
status=response.status_code
).inc()
REQUEST_DURATION.labels(
method=method,
endpoint=endpoint
).observe(duration)
ACTIVE_REQUESTS.labels(method=method, endpoint=endpoint).dec()
return response
# Expose metrics endpoint
metrics_app = make_asgi_app()
app.mount("/metrics", metrics_app)
# Prometheus scrapes: GET http://localhost:8000/metrics
# BAD: Alert fires constantly during deploys
- alert: HighCPU
expr: node_cpu_seconds_total > 80
for: 1m # Too short!
# GOOD: Alert with context and reasonable threshold
- alert: SustainedHighCPU
expr: |
(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)) > 80
for: 10m # Grace period
annotations:
description: "CPU >80% for 10 minutes on {{ $labels.instance }}"
# GOOD Dashboard Hierarchy:
1. Executive Dashboard (business metrics, SLOs)
2. Service Dashboard (RED metrics: Rate, Errors, Duration)
3. Resource Dashboard (CPU, memory, disk, network)
4. Debug Dashboard (detailed metrics for troubleshooting)
# BAD: 50 panels on one dashboard with no organization
# Grafana AI Observability features:
# 1. Anomaly Detection
# Automatically detects unusual patterns in metrics
# (configured in Grafana UI, not code)
# 2. Predictive Alerts
# ML models predict future resource exhaustion
- alert: PredictedDiskFull
expr: predict_linear(node_filesystem_avail_bytes[1h], 4*3600) < 0
annotations:
summary: "Disk will be full in 4 hours"
# 3. Root Cause Analysis
# Grafana Correlate plugin finds related metrics during incidents
# 4. AIOps Recommendations
# Suggests optimal alert thresholds based on historical data