| name | comprehensive-visual-extraction |
| description | 2026 SOTA skill for exhaustive, hallucination-free visual extraction using Grounded Chain-of-Thought (GCoT), coordinate mapping, and visual attention re-engagement. |
| category | visual-extraction |
| version | 1.0.0 |
Comprehensive Visual Extraction (CVE) Skill - 2026 Edition
Overview
This skill equips the agent with 2026 State-of-the-Art (SOTA) methodologies for converting visual inputs into exhaustive, structured, and spatially-grounded text representations. It moves beyond legacy "describe what you see" models by enforcing Grounded Chain-of-Thought (GCoT) and precise coordinate regression. The resulting output acts as a perfect, verifiable textual proxy for the original image, designed for ingestion by other LLM agents.
Core Capabilities & 2026 Paradigm Shift
- Grounded Chain-of-Thought (GCoT): You are forbidden from stating visual facts without first anchoring them to specific spatial coordinates
[x1, y1, x2, y2]. This prevents visual hallucination.
- Native Coordinate Regression (PBD): Utilize your native multimodal capacity (or Parallel Box Decoding) to estimate bounding boxes for every extracted element, establishing a precise 2D/3D semantic map.
- Visual Attention Re-engagement (Dynamic RoI): Instead of a single static scan, logically "zoom in" on Regions of Interest (RoI), forcing a resampling of visual tokens to perfectly extract microscopic details (like exact hex colors or tiny text).
- Pixel-Level Grounding: Rely on your native capabilities to extract precise Hex/RGB color values directly from the image embedding space.
The Extraction Protocol
When invoked to analyze an image, you must strictly follow this multi-layered GCoT extraction protocol.
Phase 1: Global Grounding & Strategy
- Define the Canvas: Mentally map the image space as a coordinate plane from
[0, 0] (top-left) to [100, 100] (bottom-right). All future bounding boxes must use this percentage-based scale.
- Determine Image Topology: Classify the image (e.g., UI Mockup, Natural Scene, Diagram, Document) to establish the primary extraction strategy.
- Global Metadata Summary: Provide a 2-sentence summary of the image's overall subject, lighting, and mood.
Phase 2: The GCoT Deep Extraction Process
Generate the output using a heavily structured format. For every distinct object, text block, or UI component, you must follow the GCoT Loop.
The GCoT Loop (Mandatory for every element)
For every element you identify, follow this exact sequence:
- Anchor (Coordinate): State the estimated bounding box
[x1, y1, x2, y2].
- Identify: What is the object/element?
- Re-engage (Zoom): Explicitly state: (Zooming into RoI) to focus attention on this specific box.
- Extract Attributes:
- Color: Precise hex code (e.g.,
#FF5733) or exact shade derived from pixel-level grounding.
- Text/Data: Exact transcription (if applicable).
- State/Geometric Form: (e.g., "active button", "perfect circle").
- Spatial Relationship: Describe its geometric relationship to neighboring anchors (e.g., "Aligned along the Y-axis with Anchor
[10, 20, 30, 40]").
Categorical Breakdown (Applying the GCoT Loop)
You must apply the GCoT loop comprehensively across these categories:
- 1. Structural Layout & UI Elements: Map out the macroscopic grid. Where are the headers, sidebars, hero images, or main subjects?
- 2. Exhaustive Object Inventory (Scene Graph): Identify every discrete object, its state, and its geometric relationship to the background.
- 3. Total Text & Data Transcription: Extract all text hierarchically. Reconstruct any tables into perfect Markdown format. For charts, provide data point estimates based on coordinate mapping.
- 4. Mathematical/Geometrical Data: If equations, architectural lines, or geometric ratios are present, extract them exactly using structural parsing.
Phase 3: Spatial Verification & Validation
- The Topology Check: Review your generated anchors. Do the coordinates logically overlap or align as described? If an object is stated as "inside" another, its
[x1, y1, x2, y2] must be a subset of the parent's coordinates.
- The Blind Agent Test: Confirm that another agent, possessing only this coordinate-heavy, GCoT-structured text, could perfectly reconstruct the logical, spatial, and visual hierarchy of the image.
Output Format Specification
The final output must be formatted in clean Markdown. Use code blocks for JSON arrays if you are outputting the bounding box scene graph, and standard Markdown tables/headers for text transcription. Every major descriptive block MUST be preceded by its spatial anchor [x1, y1, x2, y2].
End of Skill Definition
