sre-optimization

name	sre-optimization
description	Bootstrap and maintain the deterministic resource-waste detector for this cluster. Trigger when: the agent is first deployed, Prometheus endpoint changes, detection thresholds need tuning, or the detector script needs repair after a failed cron run.

SRE Optimization — Detector Bootstrap

This skill guides you through creating and maintaining the deterministic resource-waste detector. The detector is a plain Python script that queries Prometheus with server-side aggregation, computes waste ratios, and outputs candidates as JSONL. YOU author it; it runs on cron without LLM involvement.

Prerequisites

Before starting, verify:

• PROMETHEUS_URL environment variable is set (or discover from kubectl get svc -n monitoring)
• kubectl is configured with the target cluster context
• /data/sre/ directory is writable
• python3 and requests library are available

Step 1: Introspect the Cluster

1. Discover the Prometheus endpoint:

   kubectl get svc -n monitoring -l app.kubernetes.io/name=prometheus
   

2. Verify connectivity:

   curl -s "$PROMETHEUS_URL/api/v1/status/build" | jq .status
   

3. Confirm required metrics exist:

   curl -s "$PROMETHEUS_URL/api/v1/query?query=count(container_cpu_usage_seconds_total)" | jq '.data.result[0].value[1]'
   curl -s "$PROMETHEUS_URL/api/v1/query?query=count(kube_pod_container_resource_requests)" | jq '.data.result[0].value[1]'
   # Raw ownership series the detector uses to map pod -> workload:
   curl -s "$PROMETHEUS_URL/api/v1/query?query=count(kube_pod_owner)" | jq '.data.result[0].value[1]'
   curl -s "$PROMETHEUS_URL/api/v1/query?query=count(kube_replicaset_owner)" | jq '.data.result[0].value[1]'
   

The detector derives pod → workload ownership from kube_pod_owner + kube_replicaset_owner directly (see the $OWNER block in references/prometheus-queries.md), so the kube-prometheus-stack recording rule namespace_workload_pod:kube_pod_owner:relabel is optional. It is fine if that rule is absent. Only if you run the Prometheus operator and prefer the rule, apply the (optional) YAML from scripts/bootstrap_detector.py and swap $OWNER for the rule name in the detector.

Step 2: Generate detector.py

Use references/detector-template.py as the base. Adapt for this cluster:

1. Verify metric label names match (some clusters use pod_name instead of pod)
2. Check if namespace filtering is needed (exclude kube-system, monitoring by default)
3. Adjust the threshold from environment variable DETECTION_THRESHOLD (default: 0.5)
4. Adjust the window from DETECTION_WINDOW (default: 14d)
5. Write the adapted script to /data/sre/detector.py

Step 3: Generate pricing.json

1. Discover node instance types:

   kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.labels.node\.kubernetes\.io/instance-type}{"\n"}{end}' | sort -u
   

2. Look up pricing for discovered instance types (or use schemas/pricing.example.json as starting point)
3. Write to /data/sre/pricing.json

Step 4: Self-Test

Run the detector in dry-run mode against a single namespace:

python3 /data/sre/detector.py \
  --prometheus-url "$PROMETHEUS_URL" \
  --pricing /data/sre/pricing.json \
  --threshold 0.5 \
  --window 14d \
  --dry-run

Verify output is valid JSONL matching schemas/candidates.schema.json.

Step 5: Register Cron

Register the detector to run every 6 hours:

Schedule: 0 */6 * * *
Command: python3 /data/sre/detector.py --prometheus-url "$PROMETHEUS_URL" --pricing /data/sre/pricing.json --output /data/sre/candidates.jsonl
Name: sre-detector

Step 6: Initialize Database

python3 scripts/bootstrap_detector.py \
  --prometheus-url "$PROMETHEUS_URL" \
  --data-dir /data/sre \
  --schema-path /data/sre/schemas/decisions.sql

Or manually:

sqlite3 /data/sre/decisions.db < /data/sre/schemas/decisions.sql

Repair Mode

If the cron run exits non-zero:

1. Read the error output from the cron log
2. Common failures:
   • Exit 1: Prometheus unreachable → check network/DNS
   • Exit 2: Malformed response → metric names may have changed; re-introspect
   • Python exception: fix the specific error in /data/sre/detector.py
3. Re-run self-test after fixing
4. Do NOT modify the cron schedule unless the fix requires a different interval

Files

• scripts/bootstrap_detector.py — One-time setup helper
• references/detector-template.py — The Python template to adapt
• references/prometheus-queries.md — PromQL patterns with documentation
• references/risk-checklist.md — Risk gates used across all skills