heliosproxy-health

name	heliosproxy-health
description	Liveness + readiness + metrics + sessions + pools + version. Every read-only observability endpoint. Use when the user says "is the proxy up", "Prometheus", "metrics", "what to alert on", "active sessions", "pool stats", or wires up a health-check probe.
allowed-tools	Bash(curl *), Read
related	["heliosproxy-overview","heliosproxy-topology","heliosproxy-shutdown"]

Health, metrics, sessions, pools

Every read-only endpoint an operator (or their alerting / load balancer) calls to know whether the proxy is alive, ready, and performing. All on the admin port (default 9090).

When to use

• Liveness / readiness probe wiring (k8s, docker-compose healthcheck)
• Prometheus scrape config
• Capacity / pool-utilization checks
• "Why is the proxy slow" first-pass triage
• Confirming a build's version + features

🔵 Read-only

Endpoint cheatsheet

Endpoint	Code	Purpose
GET /health	200	always returns {"status":"ok"} if the admin loop is running
GET /health/live	200	"alive" — same semantics as /health, separated for k8s style
GET /health/ready	200/503	200 once ≥1 backend is healthy and not draining
GET /version	200	{version, build_time} (and features where built in)
GET /metrics	200	JSON metrics — connections, queries, bytes, failovers
GET /metrics/prometheus	200	text/plain Prometheus exposition (via observability feature)
GET /sessions	200	{active_sessions: N}
GET /pools	200	per-node pool stats (active, idle, pending)

Recipes

Recipe 1: Liveness + readiness probes (k8s)

livenessProbe:
  httpGet:
    path: /health/live
    port: 9090
  initialDelaySeconds: 5
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /health/ready
    port: 9090
  initialDelaySeconds: 5
  periodSeconds: 5

Liveness must NOT use /health/ready — during a clean drain, ready flips to 503 but the process is still healthy. K8s would restart it instead of letting it drain.

Recipe 2: Quick triage one-liner

curl -s http://localhost:9090/version | jq .
curl -s http://localhost:9090/metrics | jq .
curl -s http://localhost:9090/sessions
curl -s http://localhost:9090/pools | jq .

A typical healthy snapshot:

{"version":"0.4.1","build_time":"2026-05-01T13:20:05Z"}
{"connections_accepted":17234,"connections_closed":17190,"connections_active":44,"queries_processed":982041,"bytes_received":108112334,"bytes_sent":2014223891,"failovers":0}
{"active_sessions":44}
[{"node":"pg-primary:5432","active":12,"idle":8,"pending":0},{"node":"pg-replica-1:5432","active":31,"idle":4,"pending":0}]

Recipe 3: Prometheus scrape

# prometheus.yml
scrape_configs:
  - job_name: heliosproxy
    static_configs:
      - targets: ['proxy:9090']
    metrics_path: /metrics/prometheus
    scrape_interval: 15s

Sample exported metrics:

# HELP heliosproxy_connections_active Active client connections
# TYPE heliosproxy_connections_active gauge
heliosproxy_connections_active 44
# HELP heliosproxy_queries_total Queries processed since start
# TYPE heliosproxy_queries_total counter
heliosproxy_queries_total 982041
# HELP heliosproxy_failovers_total Failovers since start
# TYPE heliosproxy_failovers_total counter
heliosproxy_failovers_total 0

The observability feature adds prometheus + opentelemetry deps — cargo install --features observability.

Recipe 4: What to alert on

Symptom	Query / metric	Severity
/health/ready 503 for >2 min	probe failure	critical
failovers_total increased	rate(heliosproxy_failovers_total[5m]) > 0	high
connections_active near [pool] max_connections	gauge ratio > 0.9	warning
pool.pending > 0 for >30 s	/pools poll	warning
queries_processed rate ≈ 0 with non-zero connections_active	drop in ingest	high
5xx from /metrics itself	proxy admin loop wedged	critical

Recipe 5: Pool utilization snapshot

curl -s http://localhost:9090/pools | jq '
  .[] | {node, used: .active, idle, pending,
         pct: (.active * 100 / (.active + .idle))}'

{"node":"pg-primary:5432","used":12,"idle":8,"pending":0,"pct":60}
{"node":"pg-replica-1:5432","used":31,"idle":4,"pending":0,"pct":89}

pct: 89 on a single replica suggests adding a replica, raising [pool] max_connections, or routing more traffic to other replicas (weight changes — see heliosproxy-config).

Recipe 6: Confirm features compiled in

curl -s http://localhost:9090/version | jq .

If the response includes a features array, it lists the compiled-in feature flags. (Older versions don't expose this — fall back to checking Cargo.toml of the build.)

If a feature-gated endpoint returns 503 (e.g. /api/replay), the feature is OFF in the build — see heliosproxy-install to rebuild with --features ha-tr.

Pitfalls

• /health always 200, even during drain. Use /health/ready for traffic-routing decisions; /health is just "the admin loop is alive."
• /metrics/prometheus requires the observability feature. Without it, the endpoint returns 404. Use /metrics (JSON) or rebuild.
• active_sessions ≠ connections_active. Sessions count client-facing connections; pool active counts backend-side connections. They diverge under multi-statement transactions or pool-mode='session' with idle clients.
• Pool stats need the pool-modes feature (default-on). Without it, /pools returns an empty array, not 503.
• Don't poll faster than [health].check_interval_secs. /topology updates only on health-check ticks; faster polling burns CPU without new data.

See also

• heliosproxy-topology — node-level health detail
• heliosproxy-shutdown — /health/ready flips during drain
• heliosproxy-anomaly — separate /anomalies endpoint, not metrics
• Code: src/admin.rs — endpoint impl
• Code: src/server.rs — metrics counters
• Demo: demos/v0.4.0/09-admin-ui/ — dashboard polls all of the above