3d-visual-grounding

name	3d_visual_grounding
description	Expert Skill for 3D Visual Grounding & Reconstruction Pipeline. Use this when user gives you a 'scene_id' and a 'query'.

Expert Skill: 3D Visual Grounding & Reconstruction Pipeline

When to use this skill

Use this skill when the user requests to find and reconstruct a specific object in a scene (video with ~300 frames) and calculate its 3d bounding box coordinates.

Instructions

You MUST strictly follow this sequential execution path:

1. Query Analysis:

   • Parse the user's natural language query to extract the target_class, visual attributes, spatial conditions, and global scene_features.

2. Initialize Image Registry:

   • Call the read tool to index all image file names from the scene video into a standard JSON list.

3. Coarse Filtering (Object Masks):

   • Apply mask-based filtering to retain only images containing the target_class.
   • Critical Check: If 0 images remain, do not proceed. Analyze if the threshold is too high and retry with a lower value. You can retry at most 3 times, if still 0 image remain, exit the procedure.

4. Fine Filtering (Scene Semantics):

   • Apply VLM-based filtering to verify if images match the scene_feature.
   • Fallback Strategy: If this filter removes ALL images, discard this step and revert to the result of Coarse Filtering (Object Masks). Use the images filtered by Coarse Filtering (Object Masks) as the images.

5. Scoring & Ranking:

   • Score the remaining images based on their alignment with the query's attributes and conditions.
   • Sort the images in descending order to prioritize the best candidates.

6. Optimal View & Target Selection:

   • Traverses the candidate images (by score) to find the most distinct observation of the scene, as the "Reference View".
   • Using VLM to identify the specific Object ID within this view that best matches the query description.

7. Reference Target Isolation:

   • Generate a clean visualization of the target object in the Reference View (isolating the specific Object ID). This serves as the visual anchor for Temporal Expansion.

8. Temporal Expansion (Video Tracking)::

   • Expand the target object search temporally from the Reference View.
   • It tracks the object frame-by-frame (forward and backward) using the VLM to verify identity and SAM to generate masks. This ensures temporal consistency and generates a continuous video clip with the target object.
   • This leverages temporal consistency to robustly collect a continuous video clip containing the target as candidate frames.

9. Multi-View Coverage Enhancement (Secondary Expansion):

   • CRITICAL STEP: A single view allows for reconstruction but may be incomplete (e.g., missing the back of the object).
   • Call the secondary expansion tool (expand_from_secondary_view) to search for a NEW high-scoring reference frame that is spatially/temporally distant from the currently collected frames.
   • If a distinct new view is found, the tool will automatically track and merge new angles into your dataset to maximize 3D coverage.
   • If this step fails, just skip it.

10. Candidate Segmentation:

    • For every verified candidate frame, generate and save the precise segmentation mask for the target object.

11. Initial 3D Reconstruction:

    • Aggregate all verified images and their corresponding masks from the temporal expansion steps to generate an initial, preliminary 3D point cloud.

12. Geometric Multi-View Expansion (Centroid Complete):

    • CRITICAL STEP: To overcome VLM blind spots and maximize multi-view coverage, call the centroid_complete tool.
    • It extracts the target's absolute 3D centroid from the initial reconstruction and mathematically projects it across other scene frames.
    • By utilizing camera extrinsics and depth-based occlusion checks (Z-Buffer), it bypasses the strict VLM scene constraints and automatically harvests point-prompted masks for all valid, unoccluded views.

13. Final Dense 3D Reconstruction:

    • Call the reconstruction tool again, but this time input the updated, comprehensive image and mask paths generated by Step 12.
    • This replaces the preliminary point cloud with a final, dense, and structurally complete PLY point cloud.

14. 3D Bounding Box Calculation:

    • Calculate the axis-aligned 3D bounding box of a point cloud file.
    • If return nan, just exit the procedure.

Strategy Tips

• Dynamic Adjustment: If a filtering step returns 0 valid images, DO NOT give up immediately. Analyze the previous Observation and consider relaxing constraints (e.g., lower threshold, skip scene filter).
• Threshold Consistency: If you decide to lower the threshold in an earlier step (e.g., from 0.5 to 0.3 in Coarse Filtering) to recover targets, you MUST maintain this lowered threshold for ALL subsequent steps (e.g., Target Identification, Candidate Segmentation). Do not revert to the default threshold, otherwise, you might filter out the target again.