stage3-scene-refinement

name

stage3-scene-refinement

description

Stage 3 orchestrator: scene-analyze-prepare (auto pre-step if needed) → auto-pass (no LLM) → planner-review (1 LLM call, revises operation plan + adds deletes) → apply revised plan → render → validation (1 LLM call) → island-refiner (needed groups only, N-iter Opus per group (default 20, configurable via num_max_iter) via stage3-sub-island-refiner) → merge-back → final render. Self-contained — auto-dispatches scene-analyze-prepare if its outputs are missing. Trigger on "/stage3-scene-refinement" or "/scene-refinement".

stage3-scene-refinement

How to run — delegation

When the user triggers /stage3-scene-refinement (or /scene-refinement), the cheapest path is to delegate the whole flow to the matching orchestrator agent:

Agent(
    description="Stage 3 refine",
    subagent_type="stage3-scene-refinement",
    prompt=f"Run Stage 3 on {scene_dir} (skip_prepare={skip_prepare}, force={force}, num_max_iter={num_max_iter})",
    run_in_background=True,   # optional — long-running
)

That agent (.claude/agents/stage3-scene-refinement.md, model=haiku) is the haiku orchestrator: it drives orchestrate.py in a --resume loop, invokes the two sub-skills (/stage3-sub-scene-analyze-prepare, /stage3-sub-scene-refiner) via the Skill tool when orchestrate.py emits awaiting_agent, and dispatches parallel Task(subagent_type="stage3-island-refiner", ...) calls — one per pending relation group — until the pipeline reports step=done. The rest of this document is the reference spec the orchestrator follows; you should not need to read it to invoke the skill.

Overview

Stage 3 pipeline. Self-contained orchestrator that auto-runs stage3-sub-scene-analyze-prepare if its outputs are missing, then produces blend/stage3-scene.blend + 5-view PNGs.

Steps: 0. Scene-analyze-prepare (auto pre-step) — check if 3 critical JSON files exist (json/object_state.json, json/blend_info.json, inputs/relation_graph.json). If missing, dispatch /stage3-sub-scene-analyze-prepare. If present, pass-through (cache hit).

1. Scene-refiner (no LLM) — heuristic + collision + graph_tool → operation_plan.json + stage3-sub-planned.blend
2. Planner-review (1 LLM call) — Opus reviews operation_plan.json against image.png + masks and writes json/operation_plan_revised.json (full-replacement plan, includes delete_object ops merged in)
3. Apply-revised-plan (deterministic) — copy blend/blender_scene.blend → blend/stage3-sub-planned.blend, run apply_plan.py on the revised plan
4. Render auto blend → render/planned/*.png
5. Validation (1 LLM call) — visual check of the planned render → json/island_groups.json
6. Rebuild islands — slice stage3-sub-planned.blend into per-group island.blend files (only groups in island_groups.json)
7. Island refinement — 20-iter Opus per group (stage3-sub-island-refiner)
8. Merge-back → stage3-scene.blend
9. Final render → render/final/*.png

Transition chain: scene-analyze-prepare → scene-refiner → planner-review → apply-revised-plan → render-auto → validation → rebuild-islands → island-refiner → merge-back → render → done

Required inputs in <scene_dir>

Stage 1 and Stage 2 outputs MUST exist. The 3 files marked scene-analyze-prepare are auto-generated by Step 0 if missing — no manual preparation needed.

File	Produced by	Auto-generated by Step 0?
image.png	stage0/stage1	no — must exist
blend/blender_scene.blend	stage2	no — must exist
inputs/object_state_annotated_mask.png	stage1	no — must exist
inputs/object_class.json	stage1	no — must exist
inputs/merge_plan.json	stage1	no — must exist
json/blender_scene.json	stage2	no — must exist
json/polygon_v2.json	stage2	no — must exist
inputs/relation_graph.json	scene-analyze-prepare	yes — Step 0 generates
json/blend_info.json	scene-analyze-prepare	yes — Step 0 generates
json/object_state.json	scene-analyze-prepare	yes — Step 0 generates

The pipeline flow

Step 0 — Scene-analyze-prepare (auto pre-step)

Before any refinement runs, orchestrate.py checks whether the 3 critical JSON files produced by stage3-sub-scene-analyze-prepare exist:

• json/object_state.json
• json/blend_info.json
• inputs/relation_graph.json

If ALL three exist and are non-empty: logs step=scene-analyze-prepare status=ok (cached) — all outputs exist and advances to Step 1 (scene-refiner) immediately.

If ANY are missing or empty: sets step_status=awaiting_agent, prints a dispatch message instructing the orchestrating agent to invoke /stage3-sub-scene-analyze-prepare <scene_dir>, and exits with code 0. After the skill produces the required outputs, the agent re-runs with --resume and Step 0 short-circuits as cached.

This makes stage3-scene-refinement self-contained — it can be invoked directly after Stage 2 without a separate /scene-analyze-prepare call. The cache check ensures zero overhead when re-running on a prepared scene.

Step 1 — Scene-refiner

Invoke /stage3-sub-scene-refiner <scene_dir>. Produces:

• blend/stage3-sub-planned.blend
• json/operation_plan.json
• json/heuristic_ops.json, json/llm_ops.json

The working file is directly named stage3-sub-planned.blend. It is later read by rebuild_islands.py and merge_islands_back.py.

Step 1.5 — Planner-review (LLM)

Subprocess: python src/run_stage3_planner_review.py --scene_dir <scene_dir> (embedded Opus vision call via claude CLI; no Agent/Skill spawn).

This step follows the same un-bypassable subprocess pattern as validation: orchestrate.py invokes the wrapper directly via subprocess.run([sys.executable, ...]), and there is no code path that produces json/operation_plan_revised.json without the real claude CLI call.

The wrapper reads the heuristic operation_plan.json together with the reference image.png, inputs/object_state_annotated_mask.png, and the auxiliary planning JSONs (json/object_state.json, json/blend_info.json, inputs/relation_graph.json, plus the heuristic plan itself). The agent prompt is the mode=planner_review section of .claude/agents/stage3-scene-planner.md. Opus reviews each operation against the image evidence and produces json/operation_plan_revised.json — a full-replacement plan that supersedes the heuristic plan AND merges in any delete_object ops for unwanted objects identified during review (removing the need for a separate identify-unwanted/apply-deletes pair). The prompt also drives four vision-evident gap scans (wall-mounted objects, ghost duplicates, off-axis facing, missing support) — categories the heuristic planner cannot detect from collision data alone, formerly covered by the retired stage3-scene-discoverer agent. A tamper-evident _planner_meta block (generated_by="run_stage3_planner_review.py", mode="planner_review", image_sha256) is written into the revised plan.

Step 1.6 — Apply-revised-plan (deterministic)

After planner-review writes json/operation_plan_revised.json, the orchestrator refreshes the working blend from the clean Stage-2 source and applies the revised plan in one shot:

cp -f <scene_dir>/blend/blender_scene.blend <scene_dir>/blend/stage3-sub-planned.blend

conda run -n sceneconductor python $SKILL/../stage3-sub-scene-refiner/src/apply_plan.py \
    "<scene_dir>/json/operation_plan_revised.json" \
    "<scene_dir>/blend/stage3-sub-planned.blend" \
    "<scene_dir>/blend/stage3-sub-planned.blend"

This step is unconditional and deterministic — no LLM involvement. The resulting stage3-sub-planned.blend supersedes the one produced in Step 1 and becomes the input to render-auto, validation, and the eventual island slicing.

Note: Island slicing (rebuild-islands) runs AFTER this step so the islands reflect the revised plan (including deletes and any LLM-corrected transforms).

Step 1.8 — Validation

Subprocess: python src/run_stage3_validation.py --scene_dir <scene_dir> (embedded Opus vision call via claude CLI; no Agent/Skill spawn).

The script visually inspects render/planned/blender_scene_view_perspective.png (rendered from the revised-plan blend) against image.png and inputs/relation_graph.json, decides which relation groups need island refinement, and writes json/island_groups.json with a _planner_meta block (generated_by="run_stage3_validation.py", mode="validation", image_sha256). Natural-language rationale is required per group — template-string values are rejected as sentinel failures.

Step 1.85 — Rebuild islands

python <repo>/.claude/skills/stage3-scene-refinement/_migrated/rebuild_islands.py <scene_dir>

Slices blend/stage3-sub-planned.blend into per-group relation_groups/<G>/island.blend files. The --use-scene-camera flag is applied internally by the updated script — no manual flag needed. Produces one island.blend per group defined in inputs/relation_graph.json.

Why after apply-revised-plan: rebuild_islands.py freezes each group's M_anchor into metadata.json, and merge_islands_back.py (Step 3) composes members as M_anchor @ M_canonical while leaving the anchor at the merge-base position. Slicing after apply-revised-plan has finalized stage3-sub-planned.blend keeps M_anchor consistent with the merge base — otherwise a revised-plan transform on an anchor would split its members from it.

Step 2 — Island-refiner (agent-dispatched, per group, sequential)

For each group_id in json/island_groups.json["groups_needing_island"], orchestrate.py sets step_status=awaiting_agent and exits with a dispatch message. The orchestrating agent then invokes the stage3-island-refiner subagent via Task tool once per pending group — each Task dispatch runs the entire 20-iter loop (default, configurable) inside a single Opus session (full context cache reuse across iters).

# Orchestrating agent invokes for each group.
# Dispatch prompt carries ONLY call-specific parameters (group path + iter count).
# All phase behavior (PATTERN/MEMBER/MICRO aggression, caps, convergence rules)
# lives in the stage3-island-refiner agent .md — the single source of truth.
Task(
    subagent_type="stage3-island-refiner",
    prompt="Refine the island at <scene_dir>/relation_groups/<G>. Run 20 iterations end-to-end."
)

Inside one Task dispatch, the subagent:

1. Runs init_iter0.py once (baseline: snapshot + render + score + empty transforms.json)
2. For N in 1..MAX_ITER (default 20): Reads iter_(N-1) renders + masked.png + prev transforms, decides per-member deltas, Writes iter_N/transforms.json with _island_meta, then Bashes iter_step.py --group-dir <DIR> --iter N (iter_step.py wraps apply_delta + render_one + score_info with sanity-check)
3. Promotes iter_{MAX_ITER}/island.blend → <group_dir>/island.blend

Groups run sequentially (GPU contention — do not parallelize Task dispatches). There is no score gating, no commit/revert — every proposal is applied as-is.

Resume detection: orchestrate.py inspects <group_dir>/simple_refiner/iter_{MAX_ITER}/transforms.json::_island_meta on --resume. If image_sha256 matches the current masked.png, the group is marked completed and skipped. Otherwise it remains pending for re-dispatch. (MAX_ITER is the value passed via --num-max-iter, default 20. group_already_done checks iter<MAX_ITER>/transforms.json for a matching _island_meta block.)

Skip any group whose relation_groups/<G>/island.blend is absent (Step 1.5 failed for that group — log and record in islands_failed, then continue).

Step 3 — Merge-back

python <repo>/.claude/skills/stage3-scene-refinement/src/merge_islands_back.py <scene_dir>

Reads inputs/relation_graph.json, finds all completed groups (those with island.blend present), and invokes _migrated/merge_island_blender_inner.py via Blender headless on a copy of blend/stage3-sub-planned.blend. Produces blend/stage3-scene.blend — the Stage-3 FINAL blend. The M_anchor @ M_canonical composition is handled inside merge_island_blender_inner.py — no new math here.

The working file is directly named stage3-scene.blend and is read by the Step 4 multi-view render.

Step 4 — Multi-view render

$BLENDER --background <scene_dir>/blend/stage3-scene.blend \
    --python <repo>/.claude/skills/general-multi-view-render/src/render_multi_view.py -- \
    --scene-dir <scene_dir> \
    --samples 256

Outputs 5 PNGs to <scene_dir>/render/. Orchestrator then copies them to render/final/:

mkdir -p <scene_dir>/render/final
cp <scene_dir>/render/blender_scene_view_*.png <scene_dir>/render/final/

State persistence

After every step, write <scene_dir>/json/stage3_state.json:

{
  "scene_dir": "<abs>",
  "step": "scene-analyze-prepare|scene-refiner|planner-review|apply-revised-plan|render-auto|validation|rebuild-islands|island-refiner|merge-back|render|done",
  "step_status": "ok|failed|running|awaiting_agent",
  "started_at": "<iso8601>",
  "updated_at": "<iso8601>",
  "scene_refiner_ops": 0,
  "islands_pending": ["G1", "G2"],
  "islands_completed": [],
  "islands_failed": [],
  "merge_back_blend": null,
  "render_outputs": []
}

Default initial step is "scene-analyze-prepare" (Step 0). Re-running with --resume picks up at the first step whose step_status is not "ok".

CLI

python orchestrate.py <scene_dir> [--islands-only] [--no-island] [--resume] [--skip-prepare] [--force] [--num-max-iter N]

Flag	Effect
--islands-only	Start at the island-refinement step — assumes planned blend and island.blends already exist
--no-island	Skip rebuild-islands + island-refiner + merge-back; render the revised-plan blend directly (fast debugging)
--resume	Resume from last incomplete step using state.json
--skip-prepare	Skip Step 0 (scene-analyze-prepare); assume its 3 outputs already exist
--force	Backup any existing Stage-3 outputs to .stage3_backup_<UTC>/ and start fresh; implicitly disables --resume; when combined with --skip-prepare, the 3 prepare outputs are preserved
--num-max-iter N	Max iter count for island-refiner per group (default 20); changing N between runs forces affected groups to re-refine

The Blender binary path is read from DIRECTORYS.yaml (blender_bin key, with platform-specific overrides for Windows / macOS). To override at the shell level, set BLENDER=/path/to/blender in the environment; there is no longer a CLI flag for this.

Arguments (user-facing)

When this skill is triggered with arguments, the orchestrating agent maps user-supplied options to orchestrate.py CLI flags as follows:

User option	Default	CLI flag emitted	Meaning
skip_prepare	False	--skip-prepare (only when True)	Skip Step 0 (scene-analyze-prepare). Use when the 3 prepare outputs (json/object_state.json, json/blend_info.json, inputs/relation_graph.json) are already present and you want to bypass the cache-check entirely.
force	False	--force (only when True)	If any existing Stage-3 outputs exist, move them to <scene_dir>/.stage3_backup_<UTC>/ and start from a clean state. Implicitly disables --resume. When combined with skip_prepare=True, the 3 prepare outputs are preserved; otherwise they are also backed up so prepare re-runs.
num_max_iter	20	--num-max-iter <N>	Max iter count for the island-refiner per group. The Task-dispatch prompt to stage3-island-refiner and the resume-detection logic both use this value, so a group whose simple_refiner/iter_<N>/transforms.json already has a matching _island_meta is treated as completed and skipped ("skip it if it has already been applied"). Changing N between runs forces affected groups to re-refine.

Invocation examples

• /stage3-scene-refinement /path/to/scene — defaults (no skip, no force, 20 iters)
• /stage3-scene-refinement /path/to/scene skip_prepare=True — assume prepare outputs already exist
• /stage3-scene-refinement /path/to/scene force=True — full reset, redo prepare and refinement
• /stage3-scene-refinement /path/to/scene force=True skip_prepare=True — reset refinement but keep prepare outputs
• /stage3-scene-refinement /path/to/scene num_max_iter=30 — run island-refiner up to 30 iters per group

Parsing rules

• Boolean args (skip_prepare, force) only emit their CLI flag when the user explicitly sets them to True. Phrases like "skip prepare", "skip-prepare", "no prepare" → skip_prepare=True. Phrases like "force re-run", "fresh", "start over", "redo from scratch" → force=True.
• num_max_iter accepts any positive integer. Phrases like "30 iterations", "max 30 iters", "30 island iters" → num_max_iter=30.
• Multiple args can be combined freely.

Failure handling

Each step is retried at most once. On second failure, surface the step name, its stderr, and the state.json snapshot, then stop. Do NOT cascade to the next step.

Files in this skill

File	Role
SKILL.md	This document
src/orchestrate.py	CLI state machine and agent dispatch coordinator
src/run_stage3_planner_review.py	Step 1.5 — subprocess wrapper: Opus LLM call for planner-review (revises operation_plan.json into operation_plan_revised.json with deletes merged)
src/run_stage3_validation.py	Step 1.8 — subprocess wrapper: Opus vision call for validation
src/run_stage3_relation_solve.py	DEAD — relation-solve step removed from pipeline; kept on disk for reference
src/merge_islands_back.py	Step 3 — thin wrapper over _migrated/merge_island_blender_inner.py
_migrated/rebuild_islands.py	Step 1.85 — slice planned blend into per-group islands
_migrated/build_island_canonical.py	Dependency of rebuild_islands — canonical-frame island builder
_migrated/make_group_masked_images.py	Dependency of rebuild_islands — produces per-group masked.png
_migrated/merge_island_blender_inner.py	Step 3 inner Blender script — M_anchor composition

Note: _migrated/ consolidates the still-active scripts from two removed skills — the original Stage-3 refine-loop and stage3-sub-scene-relation-graph's island-slicing helpers. Both source skills are deleted; this directory is now the single source of truth. Dead/legacy files from those skills have been moved to arxiv/_migrated_dead/ (7 files: merge_island.py, build_group_islands_from_graph.py, extract_scene_json.py, filter_freeze.py, render_final.py, select_groups.py, verify_island_inverse.py).

Vision step architecture — un-bypassable subprocess pattern

There are now two un-bypassable Opus subprocess steps in Stage 3: planner-review and validation. Both are embedded in subprocess wrappers (src/run_stage3_planner_review.py and src/run_stage3_validation.py) and follow the same pattern. The orchestrator (orchestrate.py) invokes each wrapper directly via subprocess.run([sys.executable, ...]) — it does NOT spawn stage3-scene-planner as an Agent or Skill. This makes the Opus vision steps structurally un-bypassable: there is no code path from orchestrate.py to json/operation_plan_revised.json or json/island_groups.json that bypasses the real claude CLI call. The same pattern is used in Stage 1 (run_mask_evaluator.py) and Stage 2 (run_stage2_director.py).

Both wrappers consume the relevant mode=... section of .claude/agents/stage3-scene-planner.md (mode=planner_review and mode=validation) as the embedded agent spec.

Step 0 dispatch architecture — sub-skill, not subprocess

Unlike the un-bypassable vision steps, Step 0 (scene-analyze-prepare) follows the agent dispatch pattern (similar to Step 1 scene-refiner):

1. orchestrate.py detects missing outputs and writes step_status=awaiting_agent to stage3_state.json
2. It prints [orchestrate:dispatch] /stage3-sub-scene-analyze-prepare <scene_dir> and exits 0
3. The orchestrating agent (Haiku) invokes the sub-skill, which produces the 3 JSON files
4. The agent re-runs orchestrate.py <scene_dir> --resume
5. Step 0 re-checks file existence, finds them now present, and short-circuits as cached

This is intentional — scene-analyze-prepare itself contains multiple sub-steps (Qwen-VL inference for object_state, Blender headless for blend_info, vision agent for relation_graph) and is best handled as an independent skill invocation rather than an opaque subprocess.

Trigger phrases

• /stage3-scene-refinement — primary trigger
• /scene-refinement — short alias

Redirect note: /scene-refine-loop, /stage3-scene-refine-loop, and /scene-relation-graph are removed. If a user invokes those, redirect: refine-loop → this skill; relation-graph → /scene-analyze-prepare (which now produces inputs/relation_graph.json inline).