refine

# Quality Refinement Loop You are orchestrating a MassGen-style quality refinement workflow. This is not a one-shot task — it is an iterative loop that produces genuinely high-quality output through structured evaluation and improvement. ## Autonomy **Default: fully autonomous.** Do not ask the user for confirmation at any step — proceed through the entire workflow without stopping. Generate criteria, produce the answer, evaluate, iterate, all without pausing for approval. Do NOT use AskUserQuestion during the refinement loop — it blocks execution in non-interactive environments (e.g. Docker sandbox). Make your best judgment and proceed. The user can override this at invocation time: - `--interactive` or `--confirm-criteria`: pause for criteria confirmation - `--confirm-each-round`: pause before each iteration - If the user says "stop at checkpoints" or "ask me before iterating", honor that Otherwise: execute the full loop autonomously until converged or max rounds hit. ## Workflow ### Phase 1: Setup 1. Accept the user's query and any prior answer or deliverable path. 2. **Git repo check** — the launch script handles git init, but verify: ```bash git rev-parse --is-inside-work-tree ``` If this fails, warn the user to relaunch via `launch-claude-tmux.sh`. 3. If you plan to use `read_media` for verification (visual/audio deliverables), create a `CONTEXT.md` in the workspace root with a brief task description. Required by the vision/audio API. Not needed for `generate_media`. 4. Call `init_session` (quality-tools MCP) with an optional label describing the task. This creates a new timestamped session directory for isolated state. 5. Generate 3-7 evaluation criteria appropriate for the task. - If the user provided custom criteria, use those instead. - Call `generate_eval_criteria` (quality-tools MCP) to store them. - Do NOT ask for confirmation unless the user requested `--interactive`. ### Phase 2: Answer Production Produce or improve the deliverable. Core principles during production: **Output-first verification**: Do not just write — run, view, interact: 1. Run/view the output 2. Interact as a user would (click buttons, navigate, test inputs) 3. Identify gaps from the interaction 4. Fix and enhance 5. Re-run and re-interact 6. Repeat until excellent Classify the deliverable by behavior to choose verification: - **Static** (document, image): Render and view with read_media - **Animated** (motion graphics): Record video, review full sequence - **Interactive** (app, website): Use it — click all buttons, navigate, test states - **Produces output** (script, tool): Test with varied and edge-case inputs - **Makes sound** (audio): Listen to actual content **Question the choices**: Don't just improve execution — question fundamental choices. Is this the right approach, or just the first? Would a different direction produce a higher quality ceiling? **Material improvement gate**: Classify planned changes as TRANSFORMATIVE, STRUCTURAL, or INCREMENTAL. If only INCREMENTAL changes remain, the approach may need to change — not the polish level. **Builder agent — scope and parallelism:** Before spawning a builder, define: - ONE cohesive deliverable (one file, one feature, one surface) - ONE defect family or architectural problem - Will the builder need runtime feedback? (if yes, work inline instead) **CRITICAL: commit before spawning builders.** Builders run in git worktrees cloned from the current commit. If your files aren't committed, the worktree starts empty and builders can't see the existing code. Stage and commit all current work before spawning any builder agents. If the work has independent pieces, spawn MULTIPLE builders in parallel (invoke them as separate Agent tools in a single turn). Each runs in its own worktree. You merge outputs after all complete. Examples of good parallel splits: - Builder 1: Rewrite the hero section (layout spec) - Builder 2: Implement the contact form (API spec) - Builder 3: Build the responsive nav (mobile spec) Do NOT bundle unrelated fixes into one builder. The builder will report `split required` if over-scoped — that's feedback, not failure. Work INLINE if: changes are incremental, well-scoped within current context, or coupled to other in-progress work. **Background media operations**: For slow media operations (video generation, batch image analysis), spawn the `media-worker` agent in background. It runs the MCP tool and writes results to the session's `media_results/` directory. Continue working while it runs — check results when needed by reading the output file. **Keep iterating within Phase 2** until the deliverable feels genuinely ready. This is a tight inner loop: build → verify (read_media, test, interact) → identify gaps → fix → re-verify. Do not rush to Phase 3. The regression guard is the final gate, not the improvement mechanism. If you're still finding issues through verification, stay in Phase 2. ### Phase 3: Pre-Submission Gate When you believe the deliverable is ready — not just "done" but "a demanding user would be satisfied": 1. Spawn the **regression-guard** agent with: - Previous version (Answer A) — if this is a revision - Current version (Answer B / candidate) - Evaluation criteria 2. Read `verdict.json` from the regression-guard's worktree: - **pass**: Proceed to Phase 4 (submit) - **fail**: Go back to Phase 2 — address the regressions, then re-check - **mixed**: Present tradeoffs to user, let them decide 3. If this is the first iteration (no prior answer), skip regression guard. ### Phase 4: Submit Answer Call `new_answer` (workflow-tools MCP) to submit the current deliverable. ### Phase 5: Evaluation (Parallel) Choose the evaluation method: #### Path A: Default single-model evaluation If no `--eval-backends` flag and no `.massgen-quality/eval_config.json`: 1. **round-evaluator** agent (foreground): - Task: evaluate the deliverable against criteria - Input: deliverable path, criteria, any prior evaluation history - Output: `verdict.json`, `critique_packet.md`, `next_tasks.json` 2. **execution-trace-analyzer** agent (foreground): - Task: analyze how the answer production phase went - Input: session transcript path from `.massgen-quality/environment.json` - Output: `process_report.md`, `process_verdict.json` Wait for both to complete. #### Path B: Multi-model evaluation via team step mode (default multi-model) If `--eval-backends` is specified OR `.massgen-quality/eval_config.json` exists with `"eval_mode": "massgen"` — AND `--massgen-orchestrated` flag is NOT set and `orchestrator` is not `"massgen"` in eval_config.json: Claude orchestrates N massgen step processes directly, one per backend model. Each backend independently evaluates the deliverable. Claude detects consensus and extracts the winning evaluation. Total parallel tasks: **N + 1** (N round-evaluator step processes + 1 execution-trace-analyzer Agent). ##### Pre-checks 1. Read `.massgen-quality/environment.json` — is `massgen.available` true? 2. Check `api_keys` in `environment.json` — are the requested backends authenticated? - Map: `openai`/`codex` → `openai`, `gemini` → `google`, `claude`/`claude_code` → `anthropic`, `grok` → `xai` 3. Verify `--step` support (read `plugin_dir` from `environment.json`): ```bash bash "<plugin_dir>/scripts/run-massgen.sh" --help 2>&1 | grep -q "\-\-step" ``` 4. If massgen unavailable, `--step` unsupported, or ALL backends lack auth: **fall back to Path A** (warn user) ##### Short name expansion | Short name | Expands to | |-----------|-----------| | `codex` | `codex/gpt-5.4` | | `gemini` | `gemini/gemini-3.1-pro-preview` | | `claude` | `claude/claude-opus-4-6` | | `grok` | `grok/grok-4.20-0309-reasoning` | Full specs like `codex/gpt-5.4` are also accepted. ##### Compose the round-evaluator prompt Build a self-contained text prompt following [eval-prompt-template.md](references/eval-prompt-template.md). The prompt includes the evaluation identity, criteria, deliverable file paths (never inline content), output format instructions, and prior evaluation history (if round > 1). Write the composed prompt to `sessions/<id>/eval_team/round_NNN/eval_prompt.md`. **IMPORTANT**: Never embed deliverable content in the prompt. Reference deliverables by absolute file path and ensure `context_paths` in the massgen config includes the deliverable's parent directory. If a deliverable is not already on disk (e.g. composed in memory), write it to a file first, then reference the path. ##### Initialize eval session Create the eval session directory: ``` sessions/<id>/eval_team/round_NNN/ config.json # session_id, models, agent_mapping, max_rounds, timeout eval_prompt.md # composed round-evaluator prompt eval_criteria.json # massgen-format criteria configs/ # per-backend YAML configs agents/ # massgen writes step results here ``` Write `config.json`: ```json { "session_id": "eval_<timestamp>", "query": "evaluation", "models": ["codex/gpt-5.4", "gemini/gemini-3.1-pro-preview"], "agent_mapping": { "agent_a": "codex/gpt-5.4", "agent_b": "gemini/gemini-3.1-pro-preview" }, "max_rounds": 3, "timeout_seconds": 600 } ``` ##### Generate per-backend configs Read `plugin_dir` from `.massgen-quality/environment.json`. For each backend, generate a step mode config using the wrapper: ```bash bash "<plugin_dir>/scripts/run-massgen.sh" \ --quickstart --headless \ --config sessions/<id>/eval_team/round_NNN/configs/<agent_id>_step.yaml \ --config-backend <type> \ --config-model <model> \ --config-agent-id <agent_id> ``` Add `--config-docker` if Docker is available. Split the backend spec on `/` (e.g., `codex/gpt-5.4` → `--config-backend codex --config-model gpt-5.4`). Use absolute paths for `--config`. ##### Convert evaluation criteria to massgen format The MCP `generate_eval_criteria` tool outputs `{id, name, description, weight}`. MassGen expects `{text, category}`. Convert each criterion: ```json {"text": "<description>", "category": "must"} ``` Write the converted array to `sessions/<id>/eval_team/round_NNN/eval_criteria.json`. ##### Launch N + 1 parallel tasks Launch all tasks in a single message with multiple tool calls: **N background Bash tasks** (one per backend): ```bash bash "<plugin_dir>/scripts/run-massgen.sh" \ --step \ --session-dir "<abs_eval_session_dir>" \ --config "<abs_config_path>" \ --eval-criteria "<abs_criteria_path>" \ --automation \ "$(cat <abs_eval_prompt_path>)" ``` Each with `run_in_background: true`. Record the background task ID for each. The wrapper handles API key sourcing and massgen resolution automatically. **1 execution-trace-analyzer** agent (foreground, unchanged): - Task: analyze how the answer production phase went - Input: session transcript path from `.massgen-quality/environment.json` - Output: `process_report.md`, `process_verdict.json` ##### Collect results and detect consensus Follow the team step mode protocol (see [team-protocol.md](../team-massgen-run/references/team-protocol.md)): 1. As each background task completes (auto-notification), read `<eval_session_dir>/agents/<agent_id>/last_action.json` 2. Wait until ALL N agents have completed their step 3. Check consensus: - If all agents answered (round 1): no consensus yet, re-launch all agents - If votes exist: check `seen_steps` for stale votes, count valid votes, majority wins - If consensus reached: proceed to parsing - If no consensus and rounds remain: re-launch every agent that does not have a valid (non-stale) vote for a current answer 4. Max 3 consensus rounds. Force consensus at 600s timeout or if `--cost-budget` exceeded — use the answer with the most votes; if no votes, use the first answer submitted. 5. If only 1 backend available, use its answer directly (no consensus needed). ##### Parse winning evaluation 1. Read the consensus winner's answer from `last_action.json` → `answer_text` (or from `agents/<winner>/<step>/answer.json` → `answer`) 2. Parse the answer text for structured evaluation content (see [eval-prompt-template.md](references/eval-prompt-template.md) parsing section): - Extract `verdict.json` → write to `sessions/<id>/evaluations/round_NNN/verdict.json` - Extract `critique_packet.md` → write to `sessions/<id>/evaluations/round_NNN/critique_packet.md` - Extract `next_tasks.json` → write to `sessions/<id>/evaluations/round_NNN/next_tasks.json` 3. If parsing fails: use the raw answer as `critique_packet.md` with a default `{"schema_version": "1", "verdict": "iterate", "scores": {}}` verdict Wait for both the step mode consensus AND the execution-trace-analyzer to complete before proceeding to Phase 6. #### Path C: Multi-model evaluation via MassGen orchestrator (legacy) If `--massgen-orchestrated` flag is set OR `.massgen-quality/eval_config.json` has `"orchestrator": "massgen"`: MassGen handles all orchestration internally as a single batch process. Simpler but less controllable — subject to monitor timeouts, no error recovery, no progress visibility. **Pre-checks**: Same as Path B (massgen available, API keys, backend mapping). Fall back to Path A if unavailable. **Launch**: 1. Write the round-evaluator prompt (same identity, criteria, and deliverable file paths as Path B — never inline content) to `sessions/<id>/eval_team/round_NNN/eval_prompt.md` 2. Generate a massgen multi-agent config with the specified backends (see `/massgen-run` skill's `references/config-templates.md`). Include `context_paths` with the project root so agents can read deliverables. 3. Check Docker availability — use Docker by default, `--no-docker` to skip 4. Run in parallel: - **MassGen batch evaluation** (via Bash, `run_in_background: true`): ```bash bash "<plugin_dir>/scripts/run-massgen.sh" \ --automation \ --config <eval_config.yaml> \ "$(cat <abs_eval_prompt_path>)" ``` - **execution-trace-analyzer** agent (foreground, unchanged) - Spawn **massgen-monitor** agent (background) with the LOG_DIR 5. When massgen completes: - Read `status.json` → extract winning agent's answer (the consensus evaluation) - Parse the answer for `verdict.json`, `critique_packet.md`, `next_tasks.json` content and write to `sessions/<id>/evaluations/round_NNN/` - If parsing fails: fall back to using the raw answer as critique_packet.md with a default iterate verdict Wait for both evaluation tasks to complete before proceeding. #### eval_config.json Optional persistent config at `.massgen-quality/eval_config.json`: ```json { "eval_mode": "massgen",