Prometheus monitoring and alerting for cloud-native observability. Use when implementing metrics collection, PromQL queries, alerting rules, or service discovery. Triggers: prometheus, promql, metrics, alertmanager, service discovery, recording rules, alerting, scrape config.
Prometheus monitoring and alerting for cloud-native observability. Use when implementing metrics collection, PromQL queries, alerting rules, or service discovery. Triggers: prometheus, promql, metrics, alertmanager, service discovery, recording rules, alerting, scrape config.
Prometheus Monitoring and Alerting
Overview
Prometheus is a powerful open-source monitoring and alerting system designed for reliability and scalability in cloud-native environments.
Architecture Components
Prometheus Server: Core component that scrapes and stores time-series data
Alertmanager: Handles alerts, deduplication, grouping, routing, and notifications
Pushgateway: Allows ephemeral jobs to push metrics (use sparingly)
Exporters: Convert metrics from third-party systems to Prometheus format
# alertmanager.ymlglobal:resolve_timeout:5mslack_api_url:"https://hooks.slack.com/services/YOUR/WEBHOOK/URL"pagerduty_url:"https://events.pagerduty.com/v2/enqueue"# Template files for custom notificationstemplates:-"/etc/alertmanager/templates/*.tmpl"# Route alerts to appropriate receiversroute:group_by: ["alertname", "cluster", "service"]
group_wait:10sgroup_interval:10srepeat_interval:12hreceiver:"default"routes:# Critical alerts go to PagerDuty-match:severity:criticalreceiver:"pagerduty"continue:true# Database alerts to DBA team-match:team:databasereceiver:"dba-team"group_by: ["alertname", "instance"]
# Development environment alerts-match:env:developmentreceiver:"slack-dev"group_wait:5mrepeat_interval:4h# Inhibition rules (suppress alerts)inhibit_rules:# Suppress warning alerts if critical alert is firing-source_match:severity:"critical"target_match:severity:"warning"equal: ["alertname", "instance"]
# Suppress instance alerts if entire service is down-source_match:alertname:"ServiceDown"target_match_re:alertname:".*"equal: ["service"]
receivers:-name:"default"slack_configs:-channel:"#alerts"title:"Alert: {{ .GroupLabels.alertname }}"text:"{{ range .Alerts }}{{ .Annotations.description }}{{ end }}"-name:"pagerduty"pagerduty_configs:-service_key:"YOUR_PAGERDUTY_SERVICE_KEY"description:"{{ .GroupLabels.alertname }}"-name:"dba-team"slack_configs:-channel:"#database-alerts"email_configs:-to:"dba-team@example.com"headers:Subject:"Database Alert: {{ .GroupLabels.alertname }}"-name:"slack-dev"slack_configs:-channel:"#dev-alerts"send_resolved:true
Best Practices
Metric Naming Conventions
Follow these naming patterns for consistency:
# Format: <namespace>_<subsystem>_<metric>_<unit>
# Counters (always use _total suffix)
http_requests_total
http_request_errors_total
cache_hits_total
# Gauges
memory_usage_bytes
active_connections
queue_size
# Histograms (use _bucket, _sum, _count suffixes automatically)
http_request_duration_seconds
response_size_bytes
db_query_duration_seconds
# Use consistent base units
- seconds for duration (not milliseconds)
- bytes for size (not kilobytes)
- ratio for percentages (0.0-1.0, not 0-100)
Label Cardinality Management
DO
# Good: Bounded cardinalityhttp_requests_total{method="GET",status="200",endpoint="/api/users"}# Good: Reasonable number of label valuesdb_queries_total{table="users",operation="select"}
DON'T
# Bad: Unbounded cardinality (user IDs, email addresses, timestamps)http_requests_total{user_id="12345"}http_requests_total{email="user@example.com"}http_requests_total{timestamp="1234567890"}# Bad: High cardinality (full URLs, IP addresses)http_requests_total{url="/api/users/12345/profile"}http_requests_total{client_ip="192.168.1.100"}
Guidelines
Keep label values to < 10 per label (ideally)
Total unique time-series per metric should be < 10,000
Use recording rules to pre-aggregate high-cardinality metrics
Avoid labels with unbounded values (IDs, timestamps, user input)
Recording Rules for Performance
Use recording rules to pre-compute expensive queries:
Alert on symptoms (user-facing impact), not causes
# alerts/symptom_based.ymlgroups:-name:symptom_alertsrules:# GOOD: Alert on user-facing symptoms-alert:HighErrorRateexpr:|
(
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
) > 0.05
for:5mlabels:severity:criticalteam:backendannotations:summary:"High error rate detected"description:"Error rate is {{ $value | humanizePercentage }} (threshold: 5%)"runbook:"https://wiki.example.com/runbooks/high-error-rate"-alert:HighLatencyexpr:|
histogram_quantile(0.95,
sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
) > 1
for:5mlabels:severity:warningteam:backendannotations:summary:"High latency on {{ $labels.service }}"description:"P95 latency is {{ $value }}s (threshold: 1s)"impact:"Users experiencing slow page loads"# GOOD: SLO-based alerting-alert:SLOBudgetBurnRateexpr:|
(
1 - (
sum(rate(http_requests_total{status!~"5.."}[1h]))
/
sum(rate(http_requests_total[1h]))
)
) > (14.4 * (1 - 0.999)) # 14.4x burn rate for 99.9% SLO
for:5mlabels:severity:criticalteam:sreannotations:summary:"SLO budget burning too fast"description:"At current rate, monthly error budget will be exhausted in {{ $value | humanizeDuration }}"
Cause-based alerts (use for debugging, not paging)
# alerts/cause_based.ymlgroups:-name:infrastructure_alertsrules:# Lower severity for infrastructure issues-alert:HighMemoryUsageexpr:|
(
node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes
) / node_memory_MemTotal_bytes > 0.9
for:10mlabels:severity:warning# Not critical unless symptoms appearteam:infrastructureannotations:summary:"High memory usage on {{ $labels.instance }}"description:"Memory usage is {{ $value | humanizePercentage }}"-alert:DiskSpaceLowexpr:|
(
node_filesystem_avail_bytes{mountpoint="/"}
/
node_filesystem_size_bytes{mountpoint="/"}
) < 0.1
for:5mlabels:severity:warningteam:infrastructureannotations:summary:"Low disk space on {{ $labels.instance }}"description:"Only {{ $value | humanizePercentage }} disk space remaining"action:"Clean up logs or expand disk"
Alert Best Practices
For duration: Use for clause to avoid flapping
Meaningful annotations: Include summary, description, runbook URL, impact
Proper severity levels: critical (page immediately), warning (ticket), info (log)
Actionable alerts: Every alert should require human action
Include context: Add labels for team ownership, service, environment
PromQL Examples
Rate Calculations
# Request rate (requests per second)
rate(http_requests_total[5m])
# Sum by service
sum(rate(http_requests_total[5m])) by (service)
# Increase over time window (total count)
increase(http_requests_total[1h])
# P95 latency
histogram_quantile(0.95,
sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)
# P50, P95, P99 latency by service
histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
# Average request duration
sum(rate(http_request_duration_seconds_sum[5m])) by (service)
/
sum(rate(http_request_duration_seconds_count[5m])) by (service)
Aggregation Operations
# Sum across all instances
sum(node_memory_MemTotal_bytes) by (cluster)
# Average CPU usage
avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance)
# Maximum value
max(http_request_duration_seconds) by (service)
# Minimum value
min(node_filesystem_avail_bytes) by (instance)
# Count number of instances
count(up == 1) by (job)
# Standard deviation
stddev(http_request_duration_seconds) by (service)
Advanced Queries
# Top 5 services by request rate
topk(5, sum(rate(http_requests_total[5m])) by (service))
# Bottom 3 instances by available memory
bottomk(3, node_memory_MemAvailable_bytes)
# Predict disk full time (linear regression)
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[1h], 4 * 3600) < 0
# Compare with 1 day ago
http_requests_total - http_requests_total offset 1d
# Rate of change (derivative)
deriv(node_memory_MemAvailable_bytes[5m])
# Absent metric detection
absent(up{job="critical-service"})