soak-and-rollback

name	soak-and-rollback
description	Monitor a workload after a right-sizing change merges and auto-revert on regression. Trigger when: a right-sizing PR is merged, or when a scheduled soak check fires for an active soak watermark.

Soak & Rollback Watcher

Monitor workloads after right-sizing changes merge. Compare post-change metrics against a pre-change baseline. Auto-revert on regression and record lessons to prevent repeating failed optimizations.

Activation

When a right-sizing PR merges (detected via polling or notification):

1. Query decisions.db for the matching decision (by pr_url)
2. Create a soak watermark record
3. Schedule a T+24h check

Detection methods (in order of preference):

• Webhook notification (if configured)
• Scheduled poll: gh pr list --repo $GITOPS_REPO --state merged --label sre-optimization
• Manual trigger: user asks "check soak status for {workload}"

Step 1: Record Baseline

At merge time, query Prometheus for the workload's key signals over the preceding 24 hours:

# P99 latency (if available via service mesh or instrumentation)
histogram_quantile(0.99,
  sum by (le) (
    rate(http_request_duration_seconds_bucket{namespace="{namespace}", service="{workload}"}[24h])
  )
)

# Error rate
sum(rate(http_requests_total{namespace="{namespace}", service="{workload}", code=~"5.."}[24h]))
/
sum(rate(http_requests_total{namespace="{namespace}", service="{workload}"}[24h]))

# CPU throttle ratio
sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="{namespace}", pod=~"{workload}-.*"}[24h]))
/
sum(rate(container_cpu_usage_seconds_total{namespace="{namespace}", pod=~"{workload}-.*"}[24h]))

Store in soak_watermarks:

INSERT INTO soak_watermarks (decision_id, workload, namespace, merge_time,
  baseline_latency_p99_ms, baseline_error_rate, baseline_throttle_ratio,
  check_scheduled_at, result)
VALUES (?, ?, ?, datetime('now'), ?, ?, ?, datetime('now', '+24 hours'), 'pending');

Step 2: Schedule Check

Register a one-shot cron task at T+24h:

Schedule: in 24 hours
Command: Invoke soak-and-rollback skill for watermark ID {soak_id}
Name: soak-check-{namespace}-{workload}

Step 3: Compare (at T+24h)

Precondition (idempotency): first read the watermark's result. Only proceed if it is still pending. If it is already pass, fail, or no_data (e.g. a fast-track revert already resolved this soak), the scheduled check is stale — no-op and exit. This prevents a late scheduled run from overwriting a completed verdict or opening a duplicate revert.

Use scripts/soak_watcher.py to query the same metrics for the 24h post-merge window and compare:

python3 scripts/soak_watcher.py \
  --prometheus-url "$PROMETHEUS_URL" \
  --namespace "{namespace}" \
  --workload "{workload}" \
  --baseline-start "{merge_time - 24h}" \
  --baseline-end "{merge_time}" \
  --soak-start "{merge_time}" \
  --soak-end "{merge_time + 24h}" \
  --thresholds '{"latency_pct": 10, "error_rate_pp": 0.5, "throttle_ratio": 5}'

Regression thresholds (defaults):

Metric	Threshold	Meaning
Latency p99	+10%	Post-change p99 is >10% higher than baseline
Error rate	+0.5pp	Absolute increase of 0.5 percentage points
CPU throttle	>5%	Post-change throttle ratio exceeds 5%

The script returns a verdict and a matching exit code:

• pass (exit 0) — at least one metric was evaluated and none regressed → Step 4a
• fail (exit 1) — a metric regressed beyond threshold → Step 4b
• no_data (exit 2) — none of the three metrics resolved, so nothing could be checked → Step 4c

Do NOT treat no_data as a pass. Latency/error need app or service-mesh instrumentation (http_request_duration_seconds_bucket / http_requests_total with a service label) and throttle needs cAdvisor; if all three are absent the soak proved nothing and the change must not be silently blessed.

Step 4a: Pass

If all metrics are within bounds:

1. Mark soak as passed:

   UPDATE soak_watermarks SET result = 'pass', check_completed_at = datetime('now')
   WHERE id = ?;
   

2. Record successful outcome:

   INSERT INTO outcomes (decision_id, soak_id, outcome, recorded_at,
     post_latency_p99_ms, post_error_rate, post_throttle_ratio, lesson)
   VALUES (?, ?, 'success', datetime('now'), ?, ?, ?,
     'Workload {workload_kind} {workload} tolerates reduction from {old} to {new}');
   

3. Comment on the PR: "Soak passed. No regressions detected in 24h post-merge."

Step 4b: Fail

If ANY metric regresses beyond threshold:

1. Open an auto-revert PR that restores the original resource values:

   • Branch: sre/revert/{namespace}-{workload}-{date}
   • Title: revert(right-size): {namespace}/{workload} — soak regression detected
   • Body: include which metrics regressed, by how much, and baseline vs post values
   • Label: sre-optimization,auto-revert,urgent

2. Mark soak as failed:

   UPDATE soak_watermarks SET result = 'fail', check_completed_at = datetime('now')
   WHERE id = ?;
   

3. Record failed outcome with lesson:

   INSERT INTO outcomes (decision_id, soak_id, outcome, recorded_at,
     post_latency_p99_ms, post_error_rate, post_throttle_ratio, lesson, veto_added)
   VALUES (?, ?, 'reverted', datetime('now'), ?, ?, ?,
     '{workload_kind} {workload} does NOT tolerate {resource} reduction below {value}', 1);
   

4. Comment on the original PR: "Soak FAILED. Revert PR opened: {revert_pr_url}"

Step 4c: No Data (escalate — do not auto-pass)

If the verdict is no_data (none of latency, error rate, or throttle resolved):

1. Do NOT mark the soak as passed and do NOT close it silently.
2. Mark it for human attention:

   UPDATE soak_watermarks SET result = 'no_data', check_completed_at = datetime('now')
   WHERE id = ?;
   

3. Comment on the PR: "Soak INCONCLUSIVE — no latency/error/throttle metrics were available for {namespace}/{workload}, so no regression check could run. A human should confirm the change is healthy or revert it."
4. Notify the user. Note which signals were missing and how to enable one (e.g. the CPU throttle metric container_cpu_cfs_throttled_seconds_total from cAdvisor is the lowest-friction option and is workload-agnostic).

This commonly affects gRPC or non-instrumented services (e.g. the Online Boutique demo), which do not export http_requests_total. Prefer the throttle signal there.

Step 5: Learn

After a failure:

1. Append to /data/sre/vetoes.md:

   ## Workload: {namespace}/{workload}
   - safety_factor: 2.0
   - Reason: Reverted on {date}. {metric} regressed by {amount} after reducing
     {resource} from {old} to {new}. Original waste ratio was {ratio}.
   

2. Future investigations for this workload class will use the tighter safety factor automatically (checked in Step 3 of right-size-investigation).

Cascading Conservatism

After a rollback, the system becomes more conservative for that workload class:

• Safety factor increases from 1.5x to 2.0x for the specific workload
• If the same workload class (e.g., all redis-*) fails twice, the entire class gets the tighter factor
• This is self-correcting: the system learns from its mistakes without human tuning

Fast-Track Revert (incident during the soak window)

The T+24h check is the routine path. If, before that check fires, an incident makes it clear a recently merged right-sizing change caused a regression (e.g. the workload starts OOMKilling or CPU-throttling right after merge), don't wait out the window — revert immediately:

1. Open the auto-revert PR now (Step 4b), citing the incident as the trigger.
2. Mark the soak watermark failed right away:

   UPDATE soak_watermarks SET result = 'fail', check_completed_at = datetime('now')
   WHERE id = ?;
   

3. Record the outcome and the lesson (Step 5) so the workload class gets the tighter safety factor.
4. Cancel the pending T+24h soak check for this workload (the watermark is no longer pending, so even if the scheduled check still fires, the Step 3 precondition makes it a no-op — but cancel it to avoid the wasted run).

Only fast-track when the evidence ties the regression to this change; unrelated incidents are out of scope for this skill.

Files

• scripts/soak_watcher.py — Deterministic metric comparison script