prometheus

name	prometheus
description	Query Prometheus API for cluster metrics, alerts, and observability data. Use when investigating cluster health, performance issues, resource utilization, or alert status. Triggers on questions like "what's the CPU usage", "show me firing alerts", "check memory pressure", "query prometheus for", or any PromQL-related requests.
user-invocable	false

Prometheus Querying

Setup

Prometheus and Alertmanager are behind OAuth2 Proxy on the internal gateway. DNS URLs (https://prometheus.internal.tomnowak.work) redirect to an OAuth login page and cannot be used for API queries via curl.

Access Methods

Option 1: kubectl exec (quick, no setup)

# Query Prometheus API directly inside the pod
kubectl --context <cluster> exec -n monitoring prometheus-kube-prometheus-stack-0 -c prometheus -- \
  wget -qO- 'http://localhost:9090/api/v1/query?query=up' | jq '.data.result'

# Query Alertmanager API
kubectl --context <cluster> exec -n monitoring alertmanager-kube-prometheus-stack-0 -c alertmanager -- \
  wget -qO- 'http://localhost:9093/api/v2/alerts' | jq .

Option 2: Port-forward (for scripts and repeated queries)

kubectl --context <cluster> port-forward -n monitoring svc/prometheus-operated 9090:9090 &
export PROMETHEUS_URL=http://localhost:9090