right-size-investigation

name	right-size-investigation
description	Investigate a right-sizing candidate flagged by the detector. Trigger when: new entries appear in /data/sre/candidates.jsonl that have not yet been investigated, or when the agent is explicitly asked to evaluate a specific workload's resource allocation.

Right-Size Investigation

Investigate a single candidate from candidates.jsonl, gather context, apply risk gates, and produce a verdict: recommend, hold, or skip.

Inputs

Read the candidate record from /data/sre/candidates.jsonl. Each record has:

• namespace, workload, workload_kind, container
• resource (cpu or memory)
• current_request, p95_usage, waste_ratio
• estimated_savings_monthly_usd

Check decisions.db to ensure this candidate hasn't already been investigated (query by namespace + workload + container + resource).

Step 1: Gather Context

# Get workload details
kubectl get {workload_kind} {workload} -n {namespace} -o yaml

# Check for an HPA targeting this workload — match by scaleTargetRef, NOT by a
# label selector. HPAs rarely carry an app={workload} label, so `-l app=...` would
# silently return nothing and the HPA HARD STOP below would never fire.
kubectl get hpa -n {namespace} -o json | \
  jq -r --arg n "{workload}" --arg k "{workload_kind}" \
  '.items[] | select(.spec.scaleTargetRef.name==$n and .spec.scaleTargetRef.kind==$k)
   | {name: .metadata.name, metrics: .spec.metrics}'

# Check for PDB
kubectl get pdb -n {namespace} -o yaml | grep -A 20 {workload}

# Get replica count
kubectl get {workload_kind} {workload} -n {namespace} -o jsonpath='{.spec.replicas}'

# Get owning team (from labels)
kubectl get {workload_kind} {workload} -n {namespace} -o jsonpath='{.metadata.labels.team}'

Query Prometheus for the 30-day peak of the workload's total usage. Run the workload-level "30-Day Peak" query from sre-optimization/references/prometheus-queries.md and read the row for this {namespace} / {workload} / {container}. That form aggregates pods to the workload inside the range, so 30 days of pod churn can't stack multiple pod peaks into an inflated total (the same reason the detector's p95 is computed that way).

Step 2: Walk Risk Checklist

Evaluate every item in sre-optimization/references/risk-checklist.md:

1. HPA targets CPU? — Using the scaleTargetRef-matched HPA from Step 1 (not a label lookup), check whether it has a CPU-utilization metric
2. Critical namespace? — Check namespace labels or known critical namespace list
3. Recent deploy? — Check rollout history for changes in last 7 days
4. Memory near OOM? — For memory candidates, check if p95 > 80% of request
5. Singleton? — replicas == 1?

Document which flags triggered and which passed.

Step 3: Check Learned Vetoes

Read /data/sre/vetoes.md for:

• Namespace-level vetoes (e.g., "never auto-optimize namespace payments")
• Workload-class vetoes (e.g., "redis-* requires 2x headroom after revert on 2024-03-15")
• Resource-type vetoes (e.g., "never reduce memory for Java workloads below 4Gi")

Also query outcomes table for previous reverts of this workload or similar workloads.

Step 4: Compute Recommendation

If no HARD STOP flags:

proposed_request = max(
    p95_usage * HEADROOM_MULTIPLIER,     # default 1.3 (env: HEADROOM_MULTIPLIER)
    peak_30d * SAFETY_FACTOR             # default 1.5 (env: SAFETY_FACTOR)
)

If SOFT flags apply, multiply factors by 1.5.

If the proposed request would save less than $5/month after adjustment, verdict = skip (not worth the change risk).

Step 5: Decision

Choose one:

• recommend — Safe to proceed with PR. All risk gates passed or only soft flags with adequate headroom.
• hold — One or more HARD STOP flags triggered. Document which ones. The human must decide.
• skip — Veto applies, savings too small, or the candidate is below threshold after context.

Step 6: Log to decisions.db

INSERT INTO decisions (candidate_id, decided_at, verdict, proposed_request,
  headroom_multiplier, risk_flags, reasoning, status)
VALUES (?, datetime('now'), ?, ?, ?, ?, ?, 'pending');

Where:

• candidate_id — match from candidates table or insert the candidate first
• risk_flags — JSON array of triggered flags
• reasoning — your full reasoning chain (what you checked, what you found, why this verdict)

Step 7: Handoff

• If verdict is recommend: invoke the right-size-pr skill next.
• If verdict is hold: notify the user with a summary of why human input is needed.
• If verdict is skip: log it and move to the next candidate.

Headroom & Safety Factors

Factor	Default	Env Override	Description
Headroom multiplier	1.3	HEADROOM_MULTIPLIER	Applied to p95 usage
Safety factor	1.5	SAFETY_FACTOR	Applied to 30-day peak
Post-revert factor	2.0	—	Used when workload class was previously reverted
Minimum savings	$5/mo	MIN_SAVINGS_USD	Skip if savings below this

These are tunable per-namespace in vetoes.md using the format:

## Namespace: payments
- headroom_multiplier: 1.8
- safety_factor: 2.0