skill-iter-tune

> **Plan tracking**: codex 无 TaskCreate/TaskUpdate/TodoWrite 任务板。进度清单用 `update_plan({ explanation?, plan: [{ step, status }] })` 维护（整体提交步骤数组，status: `pending` | `in_progress` | `completed`），权威状态始终在 session 工件中；依赖/认领（addBlockedBy/owner）是工件字段，不是工具参数。 <required_reading> @~/.maestro/workflows/run-mode.md @~/.maestro/workflows/codex-run-mode.md </required_reading> # Skill Iter Tune Iterative skill refinement through execute-evaluate-improve feedback loops. Each iteration runs the skill via Claude, evaluates output via Agy, and applies improvements via Agent. ## Architecture Overview ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ Skill Iter Tune Orchestrator (SKILL.md) │ │ → Parse input → Setup workspace → Iteration Loop → Final Report │ └────────────────────────────┬─────────────────────────────────────────────┘ │ ┌───────────────────┼───────────────────────────────────┐ ↓ ↓ ↓ ┌──────────┐ ┌─────────────────────────────┐ ┌──────────┐ │ Phase 1 │ │ Iteration Loop (2→3→4) │ │ Phase 5 │ │ Setup │ │ ┌─────┐ ┌─────┐ ┌─────┐ │ │ Report │ │ │─────→│ │ P2 │→ │ P3 │→ │ P4 │ │────→│ │ │ Backup + │ │ │Exec │ │Eval │ │Impr │ │ │ History │ │ Init │ │ └─────┘ └─────┘ └─────┘ │ │ Summary │ └──────────┘ │ ↑ │ │ └──────────┘ │ └───────────────┘ │ │ (if score < threshold │ │ AND iter < max) │ └─────────────────────────────┘ ``` ### Chain Mode Extension ``` Chain Mode (execution_mode === "chain"): Phase 2 runs per-skill in chain_order: Skill A → maestro delegate → artifacts/skill-A/ ↓ (artifacts as input) Skill B → maestro delegate → artifacts/skill-B/ ↓ (artifacts as input) Skill C → maestro delegate → artifacts/skill-C/ Phase 3 evaluates entire chain output + per-skill scores Phase 4 improves weakest skill(s) in chain ``` ## Key Design Principles 1. **Iteration Loop**: Phases 2-3-4 repeat until quality threshold, max iterations, or convergence 2. **Two-Tool Pipeline**: Claude (write/execute) + Agy (analyze/evaluate) = complementary perspectives 3. **Pure Orchestrator**: SKILL.md coordinates only — execution detail lives in phase files 4. **Progressive Phase Loading**: Phase docs read only when that phase executes 5. **Skill Versioning**: Each iteration snapshots skill state before execution 6. **Convergence Detection**: Stop early if score stalls (no improvement in 2 consecutive iterations) ## Interactive Preference Collection ```javascript // ★ Auto mode detection const autoYes = /\b(-y|--yes)\b/.test($ARGUMENTS) if (autoYes) { workflowPreferences = { autoYes: true, maxIterations: 5, qualityThreshold: 80, executionMode: 'single' } } else { const prefResponse = request_user_input({ questions: [ { question: "选择迭代调优配置：", header: "Tune Config", multiSelect: false, options: [ { label: "Quick (3 iter, 70)", description: "快速迭代，适合小幅改进" }, { label: "Standard (5 iter, 80) (Recommended)", description: "平衡方案，适合多数场景" }, { label: "Thorough (8 iter, 90)", description: "深度优化，适合生产级 skill" } ] } ] }) const configMap = { "Quick": { maxIterations: 3, qualityThreshold: 70 }, "Standard": { maxIterations: 5, qualityThreshold: 80 }, "Thorough": { maxIterations: 8, qualityThreshold: 90 } } const selected = Object.keys(configMap).find(k => prefResponse["Tune Config"].startsWith(k) ) || "Standard" workflowPreferences = { autoYes: false, ...configMap[selected] } // ★ Mode selection: chain vs single const modeResponse = request_user_input({ questions: [{ question: "选择调优模式：", header: "Tune Mode", multiSelect: false, options: [ { label: "Single Skill (Recommended)", description: "独立调优每个 skill，适合单一 skill 优化" }, { label: "Skill Chain", description: "按链序执行，前一个 skill 的产出作为后一个的输入" } ] }] }); workflowPreferences.executionMode = modeResponse["Tune Mode"].startsWith("Skill Chain") ? "chain" : "single"; } ``` ## Input Processing ``` $ARGUMENTS → Parse: ├─ Skill path(s): first arg, comma-separated for multiple │ e.g., ".claude/skills/my-skill" or "my-skill" (auto-prefixed) │ Chain mode: order preserved as chain_order ├─ Test scenario: --scenario "description" or remaining text └─ Flags: --max-iterations=N, --threshold=N, -y/--yes ``` ## Execution Flow > **⚠️ COMPACT DIRECTIVE**: Context compression MUST check update_plan phase status. > The phase currently marked `in_progress` is the active execution phase — preserve its FULL content. > Only compress phases marked `completed` or `pending`. ### Phase 1: Setup (one-time) Read and execute: `Ref: phases/01-setup.md` - Parse skill paths, validate existence - Create workspace at `{run_dir}/outputs/skill-iter-tune-{ts}/` - Backup original skill files - Initialize iteration-state.json Output: `workDir`, `targetSkills[]`, `testScenario`, initialized state ### Iteration Loop ```javascript // Orchestrator iteration loop while (true) { // Increment iteration state.current_iteration++; state.iterations.push({ round: state.current_iteration, status: 'pending', execution: null, evaluation: null, improvement: null }); // Update update_plan update_plan(iterationTask, { subject: `Iteration ${state.current_iteration}/${state.max_iterations}`, status: 'in_progress', activeForm: `Running iteration ${state.current_iteration}` }); // === Phase 2: Execute === // Read: phases/02-execute.md // Single mode: one maestro delegate call for all skills // Chain mode: sequential maestro delegate per skill in chain_order, passing artifacts // Snapshot skill → construct prompt → maestro delegate --to claude --mode write // Collect artifacts // === Phase 3: Evaluate === // Read: phases/03-evaluate.md // Construct eval prompt → maestro delegate --to agy --mode analysis // Parse score → write iteration-N-eval.md → check termination // Check termination if (shouldTerminate(state)) { break; // → Phase 5 } // === Phase 4: Improve === // Read: phases/04-improve.md // Agent applies suggestions → write iteration-N-changes.md // Update update_plan with score // Continue loop } ``` ### Phase 2: Execute Skill (per iteration) Read and execute: `Ref: phases/02-execute.md` - Snapshot skill → `iteration-{N}/skill-snapshot/` - Build execution prompt from skill content + test scenario - Execute: `maestro delegate "..." --to claude --mode write --cd "${iterDir}/artifacts"` - Collect artifacts ### Phase 3: Evaluate Quality (per iteration) Read and execute: `Ref: phases/03-evaluate.md` - Build evaluation prompt with skill + artifacts + criteria + history - Execute: `maestro delegate "..." --to agy --mode analysis` - Parse 5-dimension score (Clarity, Completeness, Correctness, Effectiveness, Efficiency) - Write `iteration-{N}-eval.md` - Check termination: score >= threshold | iter >= max | convergence | error limit ### Phase 4: Apply Improvements (per iteration, skipped on termination) Read and execute: `Ref: phases/04-improve.md` - Read evaluation suggestions - Launch general-purpose Agent to apply changes - Write `iteration-{N}-changes.md` - Update state ### Phase 5: Final Report (one-time) Read and execute: `Ref: phases/05-report.md` - Generate comprehensive report with score progression table - Write `final-report.md` - Display summary to user **Phase Reference Documents** (read on-demand when phase executes): | Phase | Document | Purpose | Compact | |-------|----------|---------|---------| | 1 | [phases/01-setup.md](phases/01-setup.md) | Initialize workspace and state | update_plan 驱动 | | 2 | [phases/02-execute.md](phases/02-execute.md) | Execute skill via maestro delegate Claude | update_plan 驱动 + 🔄 sentinel | | 3 | [phases/03-evaluate.md](phases/03-evaluate.md) | Evaluate via maestro delegate Agy | update_plan 驱动 + 🔄 sentinel | | 4 | [phases/04-improve.md](phases/04-improve.md) | Apply improvements via Agent | update_plan 驱动 + 🔄 sentinel | | 5 | [phases/05-report.md](phases/05-report.md) | Generate final report | update_plan 驱动 | **Compact Rules**: 1. **update_plan `in_progress`** → 保留完整内容，禁止压缩 2. **update_plan `completed`** → 可压缩为摘要 3. **🔄 sentinel fallback** → 若 compact 后仅存 sentinel 而无完整 Step 协议，立即 `Read()` 恢复 ## Core Rules 1. **Start Immediately**: First action is preference collection → Phase 1 setup 2. **Progressive Loading**: Read phase doc ONLY when that phase is about to execute 3. **Snapshot Before Execute**: Always snapshot skill state before each iteration 4. **Background CLI**: maestro delegate runs in background, wait for hook callback before proceeding 5. **Parse Every Output**: Extract structured JSON from CLI outputs for state updates 6. **DO NOT STOP**: Continuous iteration until termination condition met 7. **Single State Source**: `iteration-state.json` is the only source of truth ## Data Flow ``` User Input (skill paths + test scenario) ↓ (+ execution_mode + chain_order if chain mode) ↓ Phase 1: Setup ↓ workDir, targetSkills[], testScenario, iteration-state.json ↓ ┌─→ Phase 2: Execute (maestro delegate claude) │ ↓ artifacts/ (skill execution output) │ ↓ │ Phase 3: Evaluate (maestro delegate agy) │ ↓ score, dimensions[], suggestions[], iteration-N-eval.md │ ↓ │ [Terminate?]─── YES ──→ Phase 5: Report → final-report.md │ ↓ NO │ ↓ │ Phase 4: Improve (Agent) │ ↓ modified skill files, iteration-N-changes.md │ ↓ └───┘ next iteration ``` ## update_plan Pattern ```javascript // Initial state update_plan({ subject: "Phase 1: Setup workspace", activeForm: "Setting up workspace" }) update_plan({ subject: "Iteration Loop", activeForm: "Running iterations" }) update_plan({ subject: "Phase 5: Final Report", activeForm: "Generating report" }) // Chain mode: create per-skill tracking tasks if (state.execution_mode === 'chain') { for (const skillName of state.chain_order) { update_plan({ subject: `Chain: ${skillName}`, activeForm: `Tracking ${skillName}`, description: `Skill chain member position ${state.chain_order.indexOf(skillName) + 1}` }) } } // During iteration N // Single mode: one score per iteration (existing behavior) // Chain mode: per-skill status updates if (state.execution_mode === 'chain') { // After each skill executes in Phase 2: update_plan(chainSkillTask, { subject: `Chain: ${skillName} — Iter ${N} executed`, activeForm: `${skillName} iteration ${N}` }) // After Phase 3 evaluates: update_plan(chainSkillTask, { subject: `Chain: ${skillName} — Score ${chainScores[skillName]}/100`, activeForm: `${skillName} scored` }) } else { // Single mode (existing) update_plan({ subject: `Iteration ${N}: Score ${score}/100`, activeForm: `Iteration ${N} complete`, description: `Strengths: ... | Weaknesses: ... | Suggestions: ${count}` }) } // Completed — collapse update_plan(iterLoop, { subject: `Iteration Loop (${totalIters} iters, final: ${finalScore})`, status: 'completed' }) ``` ## Termination Logic ```javascript function shouldTerminate(state) { // 1. Quality threshold met if (state.latest_score >= state.quality_threshold) { return { terminate: true, reason: 'quality_threshold_met' }; } // 2. Max iterations reached if (state.current_iteration >= state.max_iterations) { return { terminate: true, reason: 'max_iterations_reached' }; } // 3. Convergence: ≤2 points improvement over last 2 iterations if (state.score_trend.length >= 3) { const last3 = state.score_trend.slice(-3); if (last3[2] - last3[0] <= 2) { state.converged = true; return { terminate: true, reason: 'convergence_detected' }; } } // 4. Error limit if (state.error_count >= state.max_errors) { return { terminate: true, reason: 'error_limit_reached' }; } return { terminate: false }; } ``` ## Error Handling | Phase | Error | Recovery | |-------|-------|----------| | 2: Execute | CLI timeout/crash | Retry once with simplified prompt, then skip | | 3: Evaluate | CLI fails | Retry once, then use score 50 with warning | | 3: Evaluate | JSON parse fails | Extract score heuristically, save raw output | | 4: Improve | Agent fails | Rollback from `iteration-{N}/skill-snapshot/` | | Any | 3+ consecutive errors | Terminate with error report | **Error Budget**: Each phase gets 1 retry. 3 consecutive failed iterations triggers termination. ## Coordinator Checklist ### Pre-Phase Actions - [ ] Read iteration-state.json for current state - [ ] Verify workspace directory exists - [ ] Check error count hasn't exceeded limit ### Per-Iteration Actions - [ ] Increment current_iteration in state - [ ] Create iteration-{N} subdirectory - [ ] Update update_plan with iteration status