| name | complex-image-editing |
| title | Beyond Simple Edits: X-Planner for Complex Instruction-Based Image Editing |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2507.05259 |
| keywords | ["Image Editing","Instruction Following","Multimodal Planning","Mask Generation","Object Localization"] |
| description | Decompose complex image editing instructions into simpler sub-tasks with automatically generated control guidance. Handles multi-object edits, preserves identity of surrounding regions, and eliminates manual mask creation. |
X-Planner: Planning-Based Image Editing from Complex Instructions
Editing images based on complex instructions requires more than direct pixel manipulation. When a user says "make the building taller and the sky more dramatic," the system must understand that these are two separate edits targeting different objects, generate precise boundaries for each, and apply appropriate transformations without bleeding into adjacent regions. X-Planner solves this by decomposing complex instructions into manageable sub-tasks, automatically generating the masks and control signals that guide editing models.
The core challenge is that complex instructions are indirectly specified and often target multiple objects. Current approaches either require users to manually provide masks or fail when identity preservation matters—editing one object corrupts its surroundings.
Core Concept
X-Planner operates as a three-stage pipeline that separates planning from execution:
- Instruction decomposition: Parse the complex instruction into simpler, atomic sub-instructions
- Mask generation: For each sub-instruction, generate precise segmentation masks tailored to the edit type
- Bounding box prediction: For insertion tasks, predict spatial locations for new objects
By treating masking as a learned task conditioned on edit type, the system generates tighter masks for texture edits and dilated masks for shape changes—each adapted to the specific editing goal.
Architecture Overview
- MLLM instruction parser: Analyzes complex instructions and produces structured sub-tasks with edit types
- Edit-specific mask generator: Creates customized segmentation masks based on edit type (replacement, style change, insertion, etc.)
- Spatial predictor: For insertions, predicts bounding boxes where new objects should appear
- Compatible editing backend: Works with existing models (UltraEdit, InstructPix2Pix, etc.)
- Iterative refinement: Applies sub-instructions sequentially, each building on previous edits
Implementation
Start by analyzing a complex instruction and decomposing it into sub-tasks:
from xplanner.decomposer import InstructionDecomposer
from xplanner.masker import MaskGenerator
decomposer = InstructionDecomposer(model="gpt-4-vision")
instruction =
sub_tasks = decomposer.decompose(
instruction=instruction,
image=image
)
