check-alerts

# Check Alerts Skill This skill helps you check and analyze Grafana alerts in the Kagenti platform. ## When to Use - User asks "what alerts are firing?" - User wants to check alert status - After platform changes or deployments - During incident investigation - When troubleshooting platform issues ## What This Skill Does 1. **List Firing Alerts**: Show all currently active alerts 2. **Alert Details**: Display alert severity, component, and description 3. **Alert History**: Check recent alert state changes 4. **Query Alert Rules**: Verify alert configuration 5. **Test Alert Queries**: Validate PromQL queries ## Examples ### Check Firing Alerts ```bash # Get all currently firing alerts from Grafana kubectl exec -n observability deployment/grafana -- \ curl -s 'http://localhost:3000/api/alertmanager/grafana/api/v2/alerts' \ -u admin:admin123 | python3 -c " import sys, json alerts = json.load(sys.stdin) firing = [a for a in alerts if a.get('status', {}).get('state') == 'active'] print(f'Firing alerts: {len(firing)}') for alert in firing: labels = alert.get('labels', {}) annotations = alert.get('annotations', {}) print(f\"\\n• {labels.get('alertname')} ({labels.get('severity')})\") print(f\" Component: {labels.get('component')}\") print(f\" Description: {annotations.get('description', 'N/A')[:100]}...\") " ``` ### List All Alert Rules ```bash # Get all configured alert rules kubectl exec -n observability deployment/grafana -- \ curl -s 'http://localhost:3000/api/v1/provisioning/alert-rules' \ -u admin:admin123 | python3 -c " import sys, json rules = json.load(sys.stdin) print(f'Total alert rules: {len(rules)}') for rule in rules: print(f\" • {rule.get('title')} ({rule.get('labels', {}).get('severity')})\") " ``` ### Check Specific Alert Configuration ```bash # Get configuration for a specific alert kubectl exec -n observability deployment/grafana -- \ curl -s 'http://localhost:3000/api/v1/provisioning/alert-rules' \ -u admin:admin123 | python3 -c " import sys, json rules = json.load(sys.stdin) alert_uid = 'prometheus-down' # Change this to the alert UID rule = next((r for r in rules if r.get('uid') == alert_uid), None) if rule: print(f\"Alert: {rule.get('title')}\") print(f\"Query: {rule.get('data', [{}])[0].get('model', {}).get('expr')}\") print(f\"noDataState: {rule.get('noDataState')}\") print(f\"execErrState: {rule.get('execErrState')}\") " ``` ### Test Alert Query Against Prometheus ```bash # Test an alert's PromQL query QUERY='up{job="kubernetes-pods",app="prometheus"} == 0' kubectl exec -n observability deployment/grafana -- \ curl -s -G 'http://prometheus.observability.svc:9090/api/v1/query' \ --data-urlencode "query=${QUERY}" | python3 -m json.tool ``` ### Check Alert Evaluation State ```bash # Check why an alert is firing or not firing kubectl exec -n observability deployment/grafana -- \ curl -s 'http://localhost:3000/api/v1/eval/rules' \ -u admin:admin123 | python3 -m json.tool ``` ## Alert Locations in Grafana UI **Access Grafana**: https://grafana.localtest.me:9443 **Credentials**: admin / admin123 **Navigation**: 1. **Alerting** → **Alert rules** - View all configured alerts 2. **Alerting** → **Alert list** - See firing/pending alerts 3. **Alerting** → **Silences** - Manage alert silences 4. **Alerting** → **Contact points** - Check notification settings 5. **Alerting** → **Notification policies** - View routing rules ## Common Alert Issues ### False Positives - Check `noDataState` configuration (should be `OK` for most alerts) - Verify query matches actual resource type (Deployment vs StatefulSet) - Test query returns correct results ### Alert Not Firing When It Should - Verify metric exists in Prometheus - Check alert threshold is appropriate - Verify `for` duration isn't too long - Check `noDataState` isn't masking the issue ### Alert Configuration Not Loading - Restart Grafana: `kubectl rollout restart deployment/grafana -n observability` - Check ConfigMap applied: `kubectl get configmap grafana-alerting -n observability` - Verify no YAML syntax errors ## Related Documentation - [Alert Runbooks](../../../docs/runbooks/alerts/) - [Alert Testing Guide](../../../docs/04-observability/ALERT_TESTING_GUIDE.md) - [CLAUDE.md Alert Monitoring](../../../CLAUDE.md#alert-monitoring) - [TODO_INCIDENTS.md](../../../TODO_INCIDENTS.md) - Current incident tracking ## Runbooks by Alert When an alert fires, consult its runbook: - `docs/runbooks/alerts/<alert-uid>.md` Example: If "Prometheus Down" alert fires → `docs/runbooks/alerts/prometheus-down.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code)