monitoring

name	monitoring
description	System monitoring and observability implementation using Prometheus, Grafana, and related tools
license	MIT
compatibility	opencode
metadata	{"audience":"devops-engineers","category":"systems-administration"}

What I do

• Configure Prometheus metrics collection
• Create Grafana dashboards
• Set up alerting rules and notifications
• Monitor application and infrastructure metrics
• Implement distributed tracing
• Configure log aggregation
• Design observability strategies
• Create SLOs and SLIs
• Analyze performance bottlenecks
• Debug production issues using metrics

When to use me

When setting up monitoring systems, creating dashboards, configuring alerts, or troubleshooting performance issues.

Core Concepts

• Prometheus metrics types (counter, gauge, histogram, summary)
• Service discovery and scraping configurations
• PromQL query language
• Grafana visualization and alerting
• Alertmanager routing and silencing
• Kubernetes metrics (kube-state-metrics, node-exporter)
• Application performance monitoring (APM)
• Distributed tracing (Jaeger, Zipkin)
• Log aggregation (ELK, Loki)
• SLO/SLI/Error budget design

Code Examples

Prometheus Configuration

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'prod-us-east-1'
    env: 'production'

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093
      timeout: 10s
      api_version: v2

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  # Prometheus self-monitoring
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Node Exporter for hardware/OS metrics
  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: '(.+):\\d+'
        replacement: '${1}'

  
    
    
        
    
        []
         
         
        []
         
         
         
        [, ]
         
         
         
         

  
    
    
        
    
        
         []

# prometheus-rules.yml
groups:
  - name: default-alerts
    rules:
      # High CPU usage
      - alert: HighCPUUsage
        expr: |
          sum(rate(container_cpu_usage_seconds_total{namespace!="kube-system"}[5m]))
          /
          sum(machine_cpu_cores) > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.node }}"
          description: "CPU usage is above 85% for more than 5 minutes"
      
      # High memory usage
      - alert: HighMemoryUsage
        expr: |
          (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is above 90% for more than 5 minutes"
      
      # Disk space low
      - alert: LowDiskSpace
        expr: |
          (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) > 0.85
         
        
           
        
           
           
      
      
        
                  
        
           
        
           
           
      
      
        
                 
           
        
           
           
      
      
        
                  
        
           
        
           
           

    
    
      
        
               
      
        
         

AlertManager Configuration

# alertmanager.yml
global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: '[e-mail]'
  smtp_auth_username: 'alertmanager'
  smtp_auth_password: '${SMTP_PASSWORD}'

route:
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'critical-pager'
      continue: true
    - match:
        severity: warning
      receiver: 'warning-slack'
      continue: true
    - match_re:
        alertname: '.*Info'
      receiver: 'informational'
      group_interval: 15m

receivers:
  - name: 'default'
    email_configs:
        
         
             
    
    
        
         
    
    
    
        
         
         
         
         
         


   
       
    
       
     [, ]

Grafana Dashboard

{
  "dashboard": {
    "title": "Application Overview",
    "uid": "app-overview",
    "tags": ["application", "overview"],
    "timezone": "browser",
    "refresh": "30s",
    "panels": [
      {
        "title": "Request Rate",
        "type": "graph",
        "gridPos": {"x": 0, "y": 0, "w": 12, "h": 8}
         
          
             
             
          
        
      
      
         
         
                
         
          
             
             
          
        
         
           
             
             
               
               
                   
                   
                   
              
            
          
        
      
      
         
         
                
         
          
             
             
          
          
             
             
          
        
      
      
         
         
                
         
          
             
             
          
          
             
             
          
        
      
    
  

Best Practices

• Use histograms for values with many distinct values (latency, sizes)
• Set appropriate recording rules for complex queries
• Implement labels consistently across metrics
• Use appropriate alert thresholds based on SLOs
• Avoid alert fatigue with proper grouping and routing
• Use service discovery instead of static configurations
• Monitor not just infrastructure but business metrics too
• Implement proper error budget policies
• Use annotations for rich alert information
• Test alerting rules before deploying to production