Master production observability with Prometheus and Grafana - the industry-standard monitoring stack for cloud-native applications. Learn metrics collection, PromQL query language, dashboard design, alerting, and AI-powered anomaly detection (Grafana AI Observability 2024).
When to Use This Skill
Monitoring production applications and infrastructure
Implementing SLOs (Service Level Objectives) and SLIs
Creating custom metrics for business KPIs
Setting up alerting for proactive incident response
Debugging performance issues with metrics analysis
Tracking API latency, error rates, and throughput
Monitoring AI/ML model performance in production
Core Principles
1. The Four Golden Signals (Google SRE)
# Always monitor these four metrics for every service:# 1. Latency - How long requests takehttp_request_duration_seconds_bucket{le="0.1",job="api"}8500http_request_duration_seconds_bucket{le="0.5",job="api"}9800http_request_duration_seconds_sum{job="api"}2450http_request_duration_seconds_count{job="api"}10000# 2. Traffic - How many requestshttp_requests_total{method="GET",status="200"}50000# 3. Errors - How many requests failhttp_requests_total{method="POST",status="500"}150# 4. Saturation - How "full" is the servicenode_memory_MemAvailable_bytes/node_memory_MemTotal_bytes<0.2
# Format: <namespace>_<subsystem>_<name>_<unit>
http_requests_total # Counter: total requests
http_request_duration_seconds # Histogram: request duration
process_cpu_seconds_total # Counter: CPU time
node_memory_bytes # Gauge: memory in bytes
# Use base units:
- seconds (not milliseconds)
- bytes (not KB/MB)
- ratio (0.0-1.0, not percentage)
Label Best Practices
# GOOD: Low-cardinality labels
http_requests_total{method="GET", status="200", endpoint="/api/users"}
# BAD: High-cardinality labels (creates too many time series)
http_requests_total{user_id="12345"} # DON'T: user_id has millions of values!
http_requests_total{ip_address="192.168.1.1"} # DON'T: IP has too many values# GOOD: Aggregate high-cardinality data
user_requests_total # Single metric without user_id label
Rule of thumb: Total cardinality = product of label values. Keep <10,000 per metric.
Recording Rules (Pre-compute Expensive Queries)
# /etc/prometheus/rules.ymlgroups:-name:api_performanceinterval:30srules:# Pre-calculate p99 latency (expensive query)-record:job:http_request_duration_seconds:p99expr:|
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le)
)
# Pre-calculate error rate-record:job:http_requests:error_rateexpr:|
sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
/ sum(rate(http_requests_total[5m])) by (job)
# Use in dashboards/alerts:# job:http_request_duration_seconds:p99 > 1.0
# BAD: Alert fires constantly during deploys-alert:HighCPUexpr:node_cpu_seconds_total>80for:1m# Too short!# GOOD: Alert with context and reasonable threshold-alert:SustainedHighCPUexpr:|
(100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)) > 80
for:10m# Grace periodannotations:description:"CPU >80% for 10 minutes on {{ $labels.instance }}"
❌ DON'T: Create dashboards without purpose
# GOOD Dashboard Hierarchy:
1. Executive Dashboard (business metrics, SLOs)
2. Service Dashboard (RED metrics: Rate, Errors, Duration)
3. Resource Dashboard (CPU, memory, disk, network)
4. Debug Dashboard (detailed metrics for troubleshooting)
# BAD: 50 panels on one dashboard with no organization
AI-Powered Observability (2024)
# Grafana AI Observability features:# 1. Anomaly Detection# Automatically detects unusual patterns in metrics# (configured in Grafana UI, not code)# 2. Predictive Alerts# ML models predict future resource exhaustion-alert:PredictedDiskFullexpr:predict_linear(node_filesystem_avail_bytes[1h],4*3600)<0annotations:summary:"Disk will be full in 4 hours"# 3. Root Cause Analysis# Grafana Correlate plugin finds related metrics during incidents# 4. AIOps Recommendations# Suggests optimal alert thresholds based on historical data
Related Skills
terraform-infrastructure: Provision Prometheus/Grafana with IaC
fastapi-web-development: Add metrics to FastAPI applications
postgresql-optimization: Monitor PostgreSQL with postgres_exporter
systematic-debugging: Use metrics to debug production issues
