loop-engineering

name

loop-engineering

description

Design, audit, or critique a closed-loop LLM agent — the propose → check → measure → feedback → refine pattern. Use whenever someone wants to add a self-critique / reflection / verifier / best-of-N / debate / self-consistency step, asks whether a refine loop will actually help, wants to optimize or debug a loop that underperforms or overspends, or asks "is this an antipattern." Triggers on phrases like "should I add a critic", "build a feedback loop for my agent", "why isn't my self-refine improving", "audit my agent loop", "best- of-N / self-consistency / reflexion / debate design", "is looping worth it here", or "what's wrong with my agent loop." Entry point is the verifier-edge decision flow; routes to a design guide, an audit checklist, and an antipatterns catalog backed by 28 digested sources.

Loop Engineering — designing & auditing closed-loop LLM agents

A practical wrapper over a curated knowledge base for the loop propose → check → measure → feedback → refine. The knowledge base lives in knowledge_base/ at the project root (01–06). This file is the entry point: it runs the one decision that governs every loop, then routes to the right deliverable.

The one principle (read this first)

A loop improves only when the checker has an edge the generator lacked — the verifier edge. Same model, same lens, re-reading its own output adds no new signal and barely helps (Self-Refine[3] is the cautionary null result). Before designing, optimizing, or defending any loop, name the edge.

The five edge kinds (strongest evidence first):

1. Tool / oracle — compiler, tests, type/formal check, real measurement, an environment that returns reward. Supported, strongest.
2. Independent sample / model — sample many and vote, or peer/debate critique. Raises reliability; creates no ground truth; needs a selector.
3. Process / step verifier — score each step, not just the final answer.
4. Adversarial lens — independent same-model instances + cross-examination supported; a different model is thin (one 20-example demo).
5. Fresh context — hypothesis only; no source isolates it. Don't budget on it.

No nameable edge → don't add the loop (R3).

Entry-point decision flow — find or manufacture the oracle

Walk top to bottom; stop at the first branch you can satisfy. (This is the load-bearing step (b) of the COMPOSE procedure in 04.)

1. Does an EXISTING HARD ORACLE decide correctness?
   (compiler, interpreter, type checker, real measurement, an environment that returns reward)
   → YES: ground the loop on it. This is the strongest case.  [R1]   e.g. CompPilot[A], PAL[9]
   → NO ↓

2. Can you MANUFACTURE A CHECKER that did not exist?         [R2]
   2a. HARD manufacture — build a pass/fail with real authority:
       generate executable tests, add a formal/type/schema check, add a retrieval/citation check.
       → real ground truth (within coverage). PREFER THIS.   e.g. AlphaCodium[12], CRITIC[22]
   2b. SOFT manufacture — build a calibrated-but-fallible score:
       train a step-verifier (PRM) or stand up a rubric/LLM judge.
       → NOT ground truth. Must be calibrated + bias-tested.  [R11]  e.g. Let's Verify[23], G-Eval[17]
   → still NOTHING to check against ↓

3. ADD INDEPENDENCE (raises reliability, creates NO ground truth)  [R3]
   sample many times and vote (Self-Consistency[5]) or run peer/debate critique (Multiagent Debate[21]).
   → This needs a SELECTOR with an edge to convert coverage to performance.  [R5]

Keep two levers distinct (R2): manufacturing a checker (2a/2b) builds a signal that did not exist; adding independence (3) multiplies attempts of the same signal — useless without a selector, and not the same as having an oracle. If the task is soft (research, writing, planning, analysis) and step 1 fails, the highest-leverage move is to manufacture the strongest check you can and climb the authority ladder — see the "Oraclizing Soft Tasks" centerpiece in 04.

Where to go next (router)

You want to…	Go to	The sharpest question it answers
Compose / design a new loop from scratch	knowledge_base/04_design_guide.md	"What checks the output, and how do I wire propose/check/measure/feedback/refine?" — incl. the COMPOSE procedure (a)–(h) and Oraclizing Soft Tasks.
Audit / optimize / debug an existing loop (underperforms, overspends, or quietly lies)	knowledge_base/05_audit_checklist.md	Graded checklist grounding → interface → control → operations → safety, plus highest-leverage fixes.
Avoid a known failure / "is this an antipattern?"	knowledge_base/06_antipatterns.md	10 named watch-outs A1–A10, each with the fix and the rule it violates.

Supporting layers (cite, don't route a user here as a starting point):

• Rules R1–R12 + source verdicts → knowledge_base/03_critique.md
• Evidence map + the 13 design axes → knowledge_base/02_patterns.md
• Per-source rows [#] / [A] → knowledge_base/01_inventory.md

Three reflexes that resolve most questions

• Designing? Before building anything: name the verifier's edge. If a single prompt or a fixed pipeline already solves the task, don't build a loop — Agentless[30] beats open-ended agents on SWE-bench Lite at $0.70/issue by not looping (R4).
• Auditing? Strip the refine step and re-measure. If the score barely moves, there was no edge — you have a scaffold, not a loop (the Self-Refine[3] null result, exactly).
• Measuring? Report reliability, not just peak. For a stochastic loop, pass@k (≥1 of k trials succeed) overstates pass^k (all k succeed) — a 75%/trial loop is only ~42% reliable over 3 trials (Demystifying Evals[20]); run multiple trials from clean, isolated environments, and separate the agent's claimed success from the environment's actual end state.

Evidence contract (honor in any answer drawn from this KB)

• Cite only the 28 digested sources: [A] (CompPilot) and [#] for #1–#25, #29, #30.
• STOP (#26), AutoGen (#27), Eureka (#28), Huang (#31) are undigested optional leads — mention if relevant, never cite as evidence.
• Keep manufacture a checker (R2) distinct from add independence (R3).
• Keep the adversarial-lens tiers distinct: independent same-model instances
  • cross-examination supported; different-model thin; fresh-context hypothesis.

Don't restate the KB from memory — open the routed file and quote its rows. If the question spans compose and audit, start at the decision flow above, then read both 04 and 05.