runtime-learning-loop

name	runtime-learning-loop
description	Design a self-improvement loop for a shipped product's AI agents — production traces feed evals, evals feed improvement proposals (prompts, playbooks, retrieval configs), and a human approval gate promotes changes with rollback. Technique-agnostic: chooses per project between ACE-style evolving playbooks, GEPA/MIPROv2 offline optimization, or simple eval-driven iteration via references/techniques.md. Load when the user asks to make my product's agents self-improving, learn from production traces, add a learning loop, evolve prompts or playbooks safely, promote agent improvements, or GEPA-style optimization. NOT harness-evolution (that improves the coding agent), NOT experimentation (product A/B tests), NOT agent-run-retro (dev-phase manual retros — this skill is the production-scale continuation).
license	MIT
metadata	{"author":"dvy1987","version":"1.0","category":"project-specific","sources":"ACE arXiv:2510.04618 (ICLR 2026), GEPA arXiv:2507.19457 (ICLR 2026 oral), DSPy MIPROv2 docs, TextGrad Nature 2025, Contextual AI ACE production lessons 2026, aegis competency-gated autonomy ladder\n","resources":{"references":["techniques.md","examples.md"]}}

Runtime Learning Loop

You design the loop that lets a shipped product's agents get measurably better from their own production activity — safely. You never assume a technique (GEPA is one option, not the default); you match technique to the project's data volume, feedback quality, and budget. The user is non-technical: present proposals as consequences, not mechanisms.

Hard Rules

Hard preconditions — refuse and route if missing: observability on the target flow (agent-observability) AND an eval harness with a held-out set (eval-pipeline). No traces + no evals = nothing to learn from and no way to know if "improvement" is real. No promotion without human approval (Apprentice mode) until the autonomy ladder's advancement gates are met — and never skip the ladder. Never optimize against the held-out set. It exists only to check proposals; touching it during optimization is self-deception. Every loop ships with: declared learnable surfaces (allowlist), rollback procedure, regression monitor, budget, and kill-switch — BEFORE the first cycle runs. Production PII never enters optimization prompts or persisted playbooks — redact at the trace layer first. Feedback design outranks algorithm choice: a multi-criteria LLM judge (relevance, groundedness, completeness, clarity) beats a bare score, and can beat ground-truth comparison (Contextual AI 2026).

Workflow

Step 1 — Precondition check (hard gate)

Verify: traces flowing on the target flow; eval harness operational; held-out split defined and quarantined; eval scores stable enough to detect the lift you're seeking. Any missing → route to agent-observability / eval-pipeline and stop.

Step 2 — Choose the technique

Read references/techniques.md (decision table: ACE-style playbook deltas for continuous online learning; GEPA for offline prompt optimization with rich textual feedback; MIPROv2 for scalar metrics + few-shot demos; TextGrad for hard single instances; manual eval-driven iteration when volume is tiny). Record choice + why in docs/learning-loop/LOOP.md.

Step 3 — Declare learnable surfaces

Allowlist exactly what the loop may change (e.g. drafter prompt, insurer playbooks, retrieval config). Everything else — code, evals, held-out data, the loop itself — is off-limits to the loop. Write the allowlist into LOOP.md.

Step 4 — Build the cycle

collect traces (sampled) → score (eval-judge, multi-criteria feedback)
→ reflect/propose (technique from Step 2) → validate on held-out set
→ promotion proposal (diff + evidence + cost) → human approves → promote
→ record (changelog + memory-capture) → monitor for regression

Proposals must show: what changes (diff), held-out before/after scores, spend so far, and rollback command. Autonomous execution: the cycle runs unattended; the human only sees hypotheses/promotions and kill-switch alerts.

Step 5 — Autonomy ladder (competency-gated)

Stage	Who promotes	Advance when
Apprentice (default)	Human approves every change	≥10 approved promotions, 0 rollbacks in last 10
Journeyman	Auto-promote behind hard safety gates (guardrail evals must pass)	≥3 months clean + regression monitor proven (caught ≥1 real regression)
Master	Relaxed gates, batched review	Only if the owner explicitly opts in
Any rollback drops the loop one stage.

Step 6 — Kill-switches and budget

Declare: max $/cycle, max cycles/week, and the ROI stop (expected remaining lift vs spend, in plain numbers — same rule as agent-run-retro). Regression monitor: if production eval scores drop below the pre-loop baseline, auto-pause the loop, alert the owner, roll back the last promotion.

Gotchas

• Reward hacking / judge gaming: optimized prompts learn to please the judge, not the user. Rotate judge prompts periodically; spot-check with human review; keep one metric the loop never sees.
• Brevity bias & context collapse (ACE findings): monolithic prompt rewrites erode detail over cycles. Prefer incremental delta updates with helpful/harmful counters over full rewrites for continuously-evolving contexts.
• Cold start: with <10 traces, seed the loop with prior context (agent purpose, data types, known failure modes, one good + one bad example) — measured +7% over traces-alone (Contextual AI 2026).
• This loop improves the PRODUCT's agents. The coding agent's harness is harness-evolution; keep stores, evals, and budgets separate.
• Offline vs online: GEPA-class optimizers are compile-time (re-run per model swap); ACE-class playbooks adapt online. Model upgrades invalidate offline-optimized prompts — re-validate after every model change.

Example

Make my appeal-drafting product self-improving from its production traces. It has Phoenix tracing and a 30-case eval set with 10 held out. Preconditions: traces ✓ eval harness ✓ held-out 10 quarantined ✓. Technique: ACE-style playbook deltas (continuous learning, no labels needed, playbooks already exist per insurer). GEPA rejected for the online path (full-rewrite latency + brevity bias on detailed playbooks); noted for one-off drafter-prompt optimization later. Learnable surfaces: drafter prompt, insurer slice playbooks. Off-limits: judges, eval sets, US-playbook (regulatory). Cycle built per Step 4; judge = 4-criteria self-eval (relevance, groundedness, completeness, clarity). Ladder: Apprentice — you approve each promotion from a diff + held-out delta + rollback command. Kill-switches: $15/cycle cap, 2 cycles/week, auto-pause on baseline regression. LOOP.md saved; logged to SKILL-OUTPUTS.

Read references/examples.md for a GEPA offline path, a regression rollback, and a refused-preconditions session.

Common Rationalizations

Excuse	Reality
"GEPA worked before, just use it"	Technique fits data + feedback shape; ACE beats GEPA for online playbook evolution, GEPA wins offline prompt compiles. Choose per project.
"Skip the held-out set, more data for learning"	Then every 'improvement' is unverifiable — the loop optimizes noise.
"Auto-promote from day one"	Apprentice gate exists because early judges are miscalibrated; earn Journeyman.
"Let the loop tune the judge too"	The loop grading itself = reward hacking by construction. Judges are off-limits surfaces.
"No budget cap, quality is priceless"	Unbounded loops burn spend on <1% lifts; the kill-switch banks wins instead.

Verification

• Preconditions verified (traces + eval harness + quarantined held-out) before any cycle
• LOOP.md contains technique choice, surface allowlist, ladder stage, budget, kill-switches, rollback
• First promotion proposal shows diff + held-out before/after + rollback command
• Regression monitor tested once (simulate a drop → loop pauses)
• docs/skill-outputs/SKILL-OUTPUTS.md appended

Red Flags

• Loop proposed without observability or evals in place
• Optimization run that touched held-out examples
• Promotion applied with no rollback path recorded
• Judge prompts inside the learnable-surface allowlist
• Autonomy stage advanced without meeting its gate

Impact Report

Learning loop: [product/flow] | Technique: [ACE-delta | GEPA | MIPROv2 | manual] — why: [1 line]
Surfaces: [allowlist] | Ladder: [stage] | Budget: [$X/cycle, N/week]
Promotions: [N approved, N rolled back] | Held-out delta: [before → after]
Kill-switch: [armed | triggered — reason] | Files: docs/learning-loop/LOOP.md