loop

name	loop
description	Hypothesis-driven iteration engine for LLM workloads — runs cycles of hypothesize, execute, review, analyze, decide, edit. Triggers: "iterate on this", "run a loop", "hypothesis-driven refinement", "improve the prompt", "tune the model".
version	3
tier	protocol

Hypothesis-driven iteration engine for any LLM workload.

Loop Protocol (The Iteration Engine)

Execute §CMD_EXECUTE_SKILL_PHASES.

ARGUMENTS: Accepts optional flags:

• --manifest <path>: Use existing manifest instead of interrogation
• --plan <path>: Skip planning, use existing LOOP_PLAN.md
• --case <path>: Focus on a single case instead of running all cases
• --continue: Resume from last iteration in current session directory

Session Parameters

{
  "taskType": "CHANGESET",
  "phases": [
    {"label": "0", "name": "Setup",
      "steps": ["§CMD_REPORT_INTENT", "§CMD_PARSE_PARAMETERS", "§CMD_SELECT_MODE", "§CMD_INGEST_CONTEXT_BEFORE_WORK"],
      "commands": [],
      "proof": ["mode", "sessionDir", "parametersParsed", "flagsParsed", "routing"],
      "gate": false},
    {"label": "1", "name": "Interrogation",
      "steps": ["§CMD_REPORT_INTENT", "§CMD_INTERROGATE"],
      "commands": ["§CMD_ASK_ROUND", "§CMD_LOG_INTERACTION"],
      "proof": ["depthChosen", "roundsCompleted", "manifestValidated"]},
    {"label": "2", "name": "Planning",
      "steps": ["§CMD_REPORT_INTENT", "§CMD_GENERATE_PLAN"],
      "commands": [],
      "proof": ["failureContext", "hypothesesRanked", "experimentsDesigned", "casesSelected", "successCriteria", "planWritten", "userApproved"]},
    {"label": "3", "name": "Calibration",
      "steps": ["§CMD_REPORT_INTENT"],
      "commands": ["§CMD_APPEND_LOG"],
      "proof": ["testFixture", "pipelineResult", "manifestSaved", "calibrationLogged"]},
    {"label": "4", "name": "Baseline",
      "steps": ["§CMD_REPORT_INTENT"],
      "commands": ["§CMD_APPEND_LOG"],
      "proof": ["casesExecuted", "baselineMetrics", "baselinePresented", "userApproved"]},
    {"label": "5", "name": "Iteration Loop",
      "steps": ["§CMD_REPORT_INTENT"],
      "commands": ["§CMD_APPEND_LOG", "§CMD_TRACK_PROGRESS"],
      "proof": ["iterationsCompleted", "logEntries", "exitCondition"]},
    {"label": "6", "name": "Synthesis",
      "steps": ["§CMD_REPORT_INTENT", "§CMD_RUN_SYNTHESIS_PIPELINE"], "commands": [], "proof": [], "gate": false},
    {"label": "6.1", "name": "Checklists",
      "steps": ["§CMD_VALIDATE_ARTIFACTS", "§CMD_RESOLVE_BARE_TAGS", "§CMD_PROCESS_CHECKLISTS"], "commands": [], "proof": [], "gate": false},
    {"label": "6.2", "name": "Debrief",
      "steps": ["§CMD_GENERATE_DEBRIEF"], "commands": [], "proof": ["debriefFile", "debriefTags"], "gate": false},
    {"label": "6.3", "name": "Pipeline",
      "steps": ["§CMD_MANAGE_DIRECTIVES", "§CMD_PROCESS_DELEGATIONS", "§CMD_DISPATCH_APPROVAL", "§CMD_CAPTURE_SIDE_DISCOVERIES", "§CMD_RESOLVE_CROSS_SESSION_TAGS", "§CMD_MANAGE_BACKLINKS", "§CMD_MANAGE_ALERTS", "§CMD_REPORT_LEFTOVER_WORK"], "commands": [], "proof": [], "gate": false},
    {"label": "6.4", "name": "Close",
      "steps": ["§CMD_REPORT_ARTIFACTS", "§CMD_REPORT_SUMMARY", "§CMD_SURFACE_OPPORTUNITIES", "§CMD_CLOSE_SESSION", "§CMD_PRESENT_NEXT_STEPS"], "commands": [], "proof": [], "gate": false}
  ],
  "nextSkills": ["/loop", "/test", "/implement", "/analyze", "/chores"],
  "directives": ["TESTING.md", "PITFALLS.md", "CONTRIBUTING.md"],
  "planTemplate": "assets/TEMPLATE_LOOP_PLAN.md",
  "logTemplate": "assets/TEMPLATE_LOOP_LOG.md",
  "debriefTemplate": "assets/TEMPLATE_LOOP.md",
  "requestTemplate": "assets/TEMPLATE_LOOP_REQUEST.md",
  "responseTemplate": "assets/TEMPLATE_LOOP_RESPONSE.md",
  "modes": {
    "convergence": {"label": "Convergence", "description": "Correctness-driven, co-evolve evaluator, problem-first", "file": "modes/convergence.md"},
    "precision": {"label": "Precision", "description": "Surgical iteration, isolate variables, classify changes", "file": "modes/precision.md"},
    "exploration": {"label": "Exploration", "description": "Bold changes, seek breakthroughs", "file": "modes/exploration.md"},
    "custom": {"label": "Custom", "description": "User-defined", "file": "modes/custom.md"}
  }
}

---

0. Setup

§CMD_REPORT_INTENT:

0: Iterating on ___ workload. Trigger: ___. Focus: ___. Not: ___.

§CMD_EXECUTE_PHASE_STEPS(0.0.*)

• Scope: Understand the workload, parse flags, select iteration strategy, load context.

Flag Parsing: Check for flags in the user's command:

• --manifest <path>: Skip interrogation, use existing manifest
• --plan <path>: Skip planning, use existing LOOP_PLAN.md
• --case <path>: Focus on a single case instead of running all cases
• --continue: Resume from last iteration in current session

Mode Selection (§CMD_SELECT_MODE):

On selection: Read the corresponding modes/{mode}.md file. It defines Role, Goal, Mindset, and Configuration.

On "Custom": Read ALL 3 named mode files first (modes/precision.md, modes/exploration.md, modes/convergence.md), then accept user's framing. Parse into role/goal/mindset.

Record: Store the selected mode. It configures:

• Phase 0 role (from mode file)
• Phase 5 iteration focus, hypothesis style, and success metric (from mode file)

Resume Check: Does --continue flag exist?

• If Yes:
  1. Read LOOP_LOG.md from session directory.
  2. Parse last iteration-complete or metrics entry to find iteration number.
  3. Read manifest path from log or ask user.
  4. Skip to Phase 5 (Iteration Loop) starting at iteration N+1.
• If No: Continue to manifest check.

Manifest Check: Does --manifest <path> exist?

• If Yes: Read the manifest, validate against schema, proceed to plan check.
• If No: Proceed to Phase 1 (Interrogation).

Plan Check: Does --plan <path> exist?

• If Yes: Read the plan, skip to Phase 3 (Calibration).
• If No: Proceed to Phase 2 (Planning).

---

1. Interrogation (Manifest Creation)

Build the workload manifest through guided questioning.

§CMD_REPORT_INTENT:

1: Interrogating ___ assumptions before designing experiments. ___. Focus: ___. Not: ___.

§CMD_EXECUTE_PHASE_STEPS(1.0.*)

Topics (Loop)

Standard topics for the command to draw from. Adapt to the workload -- skip irrelevant ones, invent new ones as needed.

• Workload identity -- What is this workload? What does it produce? What signals quality?
• Iteration goals -- What specific improvements are you targeting? What's "good enough"?
• Artifact paths -- Which files (prompts, schemas, configs) will be modified during iteration?
• Evaluation strategy -- How do you measure quality? External reviewer? Automated diff? Visual inspection?
• Failure patterns -- What kinds of errors are most common? Where does the LLM struggle?
• Case selection -- What input cases best represent the problem space? Edge cases?
• Domain context -- What background docs should the Composer agent receive for deep analysis?
• Resource constraints -- Cost per iteration? API rate limits? Time budget?
• Stopping conditions -- When should we stop iterating? Quality threshold? Plateau? Budget?
• Agent configuration -- Do you have existing reviewer/Composer prompts, or should we generate them?

Manifest Assembly

Within the interrogation rounds, build the manifest from these fields:

Core Configuration:

1. "What is this workload called?" -> workloadId
2. "Which files will be modified during iteration?" -> artifactPaths
3. "Where are the test input files (cases)?" -> casePaths (accept glob patterns)
4. "Do you have expected output files for comparison?" -> expectedPaths (optional)

Execution Configuration:

1. "What command runs the workload on a single case?" -> runCommand
2. "Where should output be written?" -> outputPath
3. "What command evaluates quality?" -> evaluateCommand
4. "Alternative review command?" -> reviewCommand (optional)
5. "Build command before running?" -> buildCommand (optional — run once before each iteration's run phase)
6. "Cache strategy?" -> cacheStrategy (optional — "force" recomputes everything, "smart" lets workload caching decide; default: "smart")

Agent Configuration:

1. "Composer agent prompt file?" -> agents.composer.promptFile (or auto-generate)
2. "Reviewer agent prompt file?" -> agents.reviewer.promptFile (or auto-generate)
3. "Domain context documents for the Composer?" -> domainDocs (optional)

Advanced:

1. "Max iterations?" -> maxIterations (default: 10)

Auto-Generation (Bootstrap)

If the user doesn't have existing agent prompts:

1. Read the artifact files from artifactPaths to understand the domain.
2. Read any domainDocs for additional context.
3. Draft a Composer prompt and reviewer prompt based on the domain.
4. Present drafts to the user for review and adjustment.
5. Save to agents.composer.promptFile and agents.reviewer.promptFile.

Manifest Validation

1. Construct: Build the manifest JSON from collected answers.
2. Validate: Check against MANIFEST_SCHEMA.json.
3. Present: Show the manifest to the user for confirmation.

---

2. Planning (Experiment Design)

Before iterating, design the experiment. Measure twice, cut once.

§CMD_REPORT_INTENT:

2: Planning iteration experiments for ___ workload. ___. Focus: ___. Not: ___.

§CMD_EXECUTE_PHASE_STEPS(2.0.*)

Step A: Gather Failure Context

1. If continuing from prior session: Read prior LOOP.md or LOOP_LOG.md for context.
2. If fresh: Form initial hypotheses from domain knowledge and manifest context.
3. Categorize: Group expected failure patterns by type.

Step B: Form Hypotheses

1. Analyze: What do we expect to improve? Even a benign initial hypothesis like "we expect cases to process correctly" is valid.
2. Hypothesize: For each improvement area, propose a root cause and a predicted outcome.
3. Rank: Order hypotheses by:
   • Likelihood: How confident are we this is the cause?
   • Testability: Can we isolate and test this cheaply?
   • Impact: How many cases would this fix?

Step C: Design Experiments

1. Map: Assign each hypothesis to a specific experiment.
2. Sequence: Order experiments by priority (high-impact, high-confidence first).
3. Define Changes: For each experiment, specify:
   • The exact file and section to modify
   • The current text and proposed change
   • Which cases will test this change

Step D: Select Cases

1. Focus Cases: Pick 3-5 cases that best test the hypotheses.
2. Regression Guards: Identify 2-3 passing cases that must stay passing.
3. Exclusions: Note any cases to ignore (and why).

Step E: Define Success Criteria

1. Quantitative: What quality threshold are we targeting?
2. Qualitative: What improvements do we expect to see?
3. Exit Conditions: When do we stop iterating?

---

3. Calibration (Single-Fixture Test)

Prove the manifest works before committing to the full loop.

§CMD_REPORT_INTENT:

3: Calibrating pipeline with single fixture for ___ workload. ___. Focus: ___. Not: ___.

§CMD_EXECUTE_PHASE_STEPS(3.0.*)

Step A: Select Test Fixture

1. Expand: Resolve casePaths globs to get actual file list.
2. Select: Pick the FIRST case for calibration.
3. Announce: "Running calibration with case: [path]"

Step B: Execute Pipeline (Single Fixture)

1. Run Workload: Execute runCommand with {case} substituted.
   • If Error: Log calibration failure, ask user to fix runCommand.
2. Check Output: Verify outputPath file was created.
   • If Missing: Log calibration failure, ask user to fix outputPath.
3. Run Evaluation (if configured): Execute evaluateCommand.
   • If Error: Log calibration failure, ask user to fix evaluateCommand.

Step C: Calibration Result

• If All Passed:

  1. Log calibration success to LOOP_LOG.md.
  2. Ask: "Calibration passed. Where should I save the manifest?"
  3. Write manifest to specified path (default: alongside workload code).
  4. Proceed to Phase 4.

• If Any Failed:

  1. Log calibration failure with details.
  2. Ask: "Calibration failed. What would you like to fix?"
  3. Update manifest based on user input.
  4. Loop: Return to Step B and retry (max 3 attempts).
  5. If 3 failures: Abort with "Please fix the manifest manually and re-run with --manifest <path>."

---

4. Baseline (Initial Metrics)

Establish the starting point before any iteration.

§CMD_REPORT_INTENT:

4: Running baseline for ___ workload. All cases, iteration 0. Focus: ___. Not: ___.

§CMD_EXECUTE_PHASE_STEPS(4.0.*)

Step A: Form Initial Hypothesis

1. State: The initial hypothesis (even a benign one: "We expect cases to process correctly with current configuration").
2. Predict: What do we expect the baseline to show?
3. Log: Append hypothesis entry to LOOP_LOG.md.

Step B: Run Cases

1. Expand: Resolve casePaths globs to get full case list.
2. Filter (if --case <path> specified): Reduce to just the specified case.
3. Execute: For each case:
   • Run runCommand
   • Run evaluateCommand (if configured)
   • Compare output to expectedPaths (if configured)
4. Log: Append result entry with baseline metrics.

Step C: Present Baseline

1. Report: "Baseline: X/Y cases passing (Z%)"
2. List Issues: Show which cases had problems and why (if known).
3. Present choice: "Baseline: X/Y passing. Ready to begin iteration?" / "Let me review first"

---

5. Iteration Loop (The Core Cycle)

UNDERSTAND -> DESIGN -> IMPLEMENT -> VERIFY -> CO-EVOLVE

The Problem-First Manifesto

1. Correctness over numbers — The goal is to make the artifact CORRECT, not to satisfy the evaluator. Aggregate scores are signals, not targets. A "regression" in score might mean the evaluator is wrong.
2. Collaborative design — Never reject user proposals. The user has domain knowledge the agent lacks. When the user suggests a change, the agent's job is to ask "What problem does this solve?" and work together on the solution — not to list reasons it might fail.
3. Co-evolve the evaluator — The evaluator/reviewer is itself a work-in-progress. When it flags correct behavior, fix the evaluator. When it misses real issues, strengthen it. Both artifacts improve together.
4. Qualitative over quantitative — After every run, classify failures as: real extraction error, evaluator false positive, evaluator miscalibration, or infrastructure bug. Only real errors drive changes.
5. Never revert based on numbers alone — Investigate what actually changed. A drop in aggregate score with architecturally correct changes means the evaluator needs updating, not the change needs reverting.

§CMD_REPORT_INTENT:

5: Entering iteration loop for ___ workload. ___. Focus: ___. Not: ___.

§CMD_EXECUTE_PHASE_STEPS(5.0.*)

For Each Iteration (1 to maxIterations):

Step A: UNDERSTAND (Problem-First)

1. Ask: "What is the problem we are solving?" — State the specific parsing/extraction/quality failure in concrete terms. Not "numbers dropped" but "the model doesn't separate recap totals from table content."
2. Collaborate: Present the problem to the user. Ask for their diagnosis and domain insight. The user sees things the agent cannot.
3. Hypothesize: Together with the user, form a hypothesis: "The artifact lacks [X], causing [Y] because [Z mechanism]."
4. Predict: State the expected outcome. "After this change, [specific pages/cases] should improve because [reason]."
5. Log: Append hypothesis entry to LOOP_LOG.md with problem statement and prediction.

Step B: RUN

1. Execute: Run runCommand for all cases (or focused cases per plan).
2. Collect Output: Store results at outputPath.
3. Log: Append experiment entry.

Step C: REVIEW + CLASSIFY

1. Collect data: Run evaluateCommand to gather per-case results.
   • If expectedPaths configured: also compute diff-based data.
2. Examine per-case results qualitatively BEFORE looking at aggregates. For each case, ask: what changed from baseline? Is the change correct?
3. Classify every failure (MANDATORY before drawing conclusions):
   • Real error — the artifact produced wrong output
   • Evaluator false positive — the evaluator flagged correct behavior
   • Evaluator miscalibration — the evaluator is applying wrong rules
   • Infrastructure bug — empty output, stale data, schema validation failure
4. Report by classification first: "X real errors, Y evaluator false positives, Z infrastructure bugs." Aggregate scores are secondary context.
5. Co-evolve: If evaluator false positives or miscalibrations are found, fix the evaluator NOW — don't defer. Both artifacts improve in the same iteration.
6. Log: Append critique entry with classification breakdown, then aggregate numbers as supporting context.

Step D: ANALYZE (Composer Subagent)

1. Invoke Composer: Launch the Composer subagent via Task tool with:

   • The problem statement and user diagnosis from Step A (the collaborative understanding)
   • Full prompt/schema text from artifactPaths
   • Classified evaluation results from Step C (not raw scores — the qualitative breakdown)
   • Complete iteration history (hypothesis records from LOOP_LOG.md)
   • Domain docs from domainDocs
   • The Composer prompt template from agents.composer.promptFile

2. Composer Output: The Composer produces:

   • Root Cause Analysis: Why the current artifacts produce these failures
   • Strategic Options: 3 approaches to fix the root cause
   • Recommendation: 1 recommended fix with 2 alternatives
   • Each fix must be a structural prompt engineering technique -- not a surface-level suggestion

3. Present All 3 Options: Always show the recommended fix AND both alternatives to the user.

4. Log: Append Composer analysis entry.

5. Write Iteration Report: After the Composer analysis, generate a failure mode report using assets/TEMPLATE_LOOP_REPORT.md.

   • Filename: LOOP_REPORT_NNN.md where NNN is the zero-padded iteration number (e.g., LOOP_REPORT_001.md).
   • Location: Session directory (alongside LOOP_LOG.md).
   • Content: The Composer populates all 6 sections — Iteration Summary, Failure Modes (categorized with affected cases), Root Cause Analysis, Fix Options, Regression Risks, and Case-by-Case Breakdown.
   • Case-specific: Each failing case is listed by path/name with its specific symptom and failure mode classification.
   • Announce: Link the report file to the user: "Iteration report written: [clickable path]"

Step E: DECIDE

1. Present: Show the 3 options to the user:

   • "Option 1: [Recommended fix summary]" -- Apply the recommended change
   • "Option 2: [Alternative A summary]" -- Apply alternative A
   • "Option 3: [Alternative B summary]" -- Apply alternative B
   • "Skip this iteration" -- Move to next iteration with a different hypothesis

2. On rejection handling: If the user skips or wants something different:

   • "Next iteration with new hypothesis" -- Skip to next cycle with a fresh hypothesis
   • "Retry with feedback" -- Feed your reason back to the Composer for a refined suggestion

3. Log: Append decision entry.

Step F: EDIT

1. Apply: Make the chosen edit to the artifact files.
2. Log: Append edit-applied entry with exact changes.
3. Verify Prediction: The NEXT iteration's RUN step will test the hypothesis. This is the scientific method -- the edit IS the experiment; the next run IS the measurement.

Convergence Check (End of Each Iteration)

Evaluate the iteration hypothesis-first, then use metrics as supporting evidence:

1. Did we solve the stated problem? Review Step A's problem statement and prediction.

   • If yes: Log hypothesis confirmed. Check if broader goals are met.
   • If partially: Log what worked and what didn't. Refine hypothesis for next iteration.
   • If no: Log hypothesis rejected. Form new hypothesis for next iteration.

2. Did the prediction hold? Compare Step A's specific prediction to classified results from Step C.

   • Prediction matched → hypothesis confirmed, high confidence in the change.
   • Prediction partially matched → mechanism is right but scope was off.
   • Prediction wrong → root cause analysis was incorrect, revisit understanding.

3. What do the numbers say? Use aggregate metrics as supporting evidence, not as the verdict.

   • All cases passing → converged, exit loop.
   • Max iterations reached → exit to synthesis with remaining issues documented.
   • No hypothesis confirmed for 2 iterations → present choice: different approach or accept current state.

4. Classify any unexpected changes. If cases that weren't targeted by the hypothesis changed:

   • Classify each change: real regression / evaluator miscalibration / expected side-effect / infrastructure noise.
   • NEVER revert based on numbers alone. Present the classified analysis to the user.
   • Present choice:
     • "Fix evaluator and re-measure" -- The evaluator is miscalibrated
     • "Accept tradeoff and continue" -- The improvement elsewhere outweighs the real change
     • "Try different approach" -- The change caused real regressions, form new hypothesis

5. Otherwise: Continue to next iteration (loop back to Step A).

---

6. Synthesis

When iteration is complete.

§CMD_REPORT_INTENT:

6: Synthesizing. ___ iterations completed, ___ cases passing. Focus: ___. Not: ___.

§CMD_EXECUTE_PHASE_STEPS(6.0.*)

Debrief notes (for LOOP.md):

• Populate iteration history table with hypothesis records.
• List all edits made with impact and hypothesis outcomes.
• Document remaining failures with root cause analysis.
• Capture Composer insights and generalizable learnings.
• Reference iteration reports: list all LOOP_REPORT_NNN.md files generated during the session.

Walk-through config:

§CMD_WALK_THROUGH_RESULTS Configuration:
  mode: "results"
  gateQuestion: "Iteration complete. Walk through remaining issues and recommendations?"
  debriefFile: "LOOP.md"
  templateFile: "assets/TEMPLATE_LOOP.md"

---

Appendix: Invariants

The protocol respects these invariants:

• ¶INV_HYPOTHESIS_AUDIT_TRAIL: Every iteration must produce a hypothesis record (prediction + outcome). The log is the audit trail of what was tried, predicted, and learned.
• ¶INV_REVIEW_BEFORE_COMPOSE: The Composer subagent MUST receive evaluation results as input. It never operates on raw outputs alone -- the reviewer/evaluator provides the structured quality signal.
• ¶INV_COMPOSER_STRUCTURAL_FIXES: Composer suggestions must be structural prompt engineering fixes ("add anchoring rule for table boundaries"), not surface-level ("extract the table correctly"). If a suggestion lacks a concrete mechanism, it is rejected.
• ¶INV_RE_REVIEW_AFTER_EDIT: After each edit, the next iteration's RUN+REVIEW step provides fresh evaluation. Do not compare old reviews to new outputs.
• ¶INV_EXPECTED_OPTIONAL: expectedPaths in the manifest is optional. The loop must work from evaluation critiques alone.
• ¶INV_MANIFEST_COLOCATED: Manifests live with workload code, not in a central registry.
• ¶INV_NO_SILENT_REGRESSION: Score drops are investigated and classified before any action. Never silently reverted.
• ¶INV_VALIDATE_BEFORE_ITERATE: Single-case calibration before the full loop.
• ¶INV_CORRECTNESS_OVER_SCORE: The goal is correct output, not high scores. Evaluator scores are signals, not targets. When the evaluator disagrees with correct output, fix the evaluator.
• ¶INV_CO_EVOLVE_EVALUATOR: The evaluator/reviewer is a co-evolving artifact. False positives and miscalibrations are fixed in the same iteration they're discovered, not deferred.
• ¶INV_COLLABORATIVE_DESIGN: Never reject user proposals. Ask "What problem does this solve?" and work together on the solution. The user has domain knowledge the agent lacks.
• ¶INV_CLASSIFY_BEFORE_CONCLUDE: Every failure must be classified (real error / evaluator false positive / evaluator miscalibration / infrastructure bug) before any conclusion is drawn about an iteration's outcome.