Skip to main content Home Creators adu2021 skillxiv treevgr-visual-grounded-reasoning-evidence
treevgr-visual-grounded-reasoning-evidence Train vision-language models to produce visually grounded reasoning by enforcing traceable evidence via bounding box localization, using a novel benchmark (TreeBench) and RL-based training with dual IoU rewards for both recall and precision.
Jump to install Skills Marketplace Discover and explore AI skills built by the community.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Show prompt details A direct command skips the review prompt. Inspect the source before running it.
npx skills add https://github.com/ADu2021/skillXiv --skill treevgr-visual-grounded-reasoning-evidenceThe command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
Download Zip Downloading... More from this repository
Related occupations SOC
Based on SOC occupation classification
name treevgr-visual-grounded-reasoning-evidence title Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2507.07999 keywords ["Visual Reasoning","Evidence Grounding","Reinforcement Learning","Object Localization","VQA"] description Train vision-language models to produce visually grounded reasoning by enforcing traceable evidence via bounding box localization, using a novel benchmark (TreeBench) and RL-based training with dual IoU rewards for both recall and precision.
TreeVGR: Visual Grounded Reasoning with Traceable Evidence
Standard vision-language models answer questions but rarely show their visual reasoning. TreeVGR introduces a training paradigm and benchmark that demand evidence: the model must identify and localize specific regions supporting its answers. TreeBench is a diagnostic benchmark of 405 carefully curated VQA pairs emphasizing fine perception in cluttered scenes. TreeVGR trains models to produce bounding boxes alongside answers, using reinforcement learning with dual IoU metrics that reward both recall (did you find all relevant objects?) and precision (did you avoid false positives?).
This approach transforms VQA from a classification task into a grounded reasoning task, improving explainability and robustness.
Core Concept
Visual reasoning without visual evidence is brittle. Models can guess correctly by exploiting spurious correlations. TreeVGR enforces visual grounding: to answer a question, the model must identify which objects or regions support the answer. The training reward combines three signals: (1) answer correctness (did you get it right?), (2) bounding box recall (did you localize all relevant objects?), (3) bounding box precision (did you avoid false positives?). This multi-faceted reward encourages genuine visual understanding rather than shortcut learning.
The benchmark carefully samples cluttered scenes (dense objects) and emphasizes second-order reasoning (reasoning about attributes, spatial relations, transformations) beyond simple localization.
Architecture Overview
Base Model : Qwen2.5-VL-7B vision-language modelQuestion-Guided Vision : Extract features conditioned on query, focus on relevant regionsGrounding Head : Predict bounding boxes for evidence regionsAnswer Head : Generate answer text conditioned on grounded featuresTreeBench : 405 VQA pairs with bounding box annotations in cluttered scenesDual IoU Rewards : Separate metrics for recall and precision in localization
Implementation
Step 1: Create TreeBench-Style Evaluation Dataset
Curate a dataset emphasizing fine visual perception and evidence-based reasoning. Select dense object scenes and annotate questions across two categories:
import json
import torch
from typing import List , Dict , Tuple
from dataclasses import
@dataclass
class
TreeBenchExample
str
str
str
List
Tuple
int
int
int
int
str
str
def
create_treebench_dataset
num_examples=405 , num_images=1000
"""
Create TreeBench: VQA dataset emphasizing visual perception
in dense object scenes with bounding box annotations.
"""
for
in
enumerate
"color"
f"What color is the {random_object(image)} ?"
"material"
f"What material is the {random_object(image)} ?"
"attribute"
f"Describe the {random_object(image)} ."
"spatial"
f"Where is the {random_object(image)} located?"
"count"
f"How many objects are {random_property(image)} ?"
"perspective"
f"What would you see from {random_direction(image)} ?"
"containment"
f"Is the {random_object(image)} inside or outside?"
"transformation"
f"If we rotate the {random_object(image)} 90°, where would it be?"
"relationship"
f"What is the relationship between {random_object(image)} and {random_object(image)} ?"
"inference"
f"Based on the scene, what might happen next?"
for
in
"perception"
if
in
0
for
in
else
"reasoning"
return
def
sample_dense_object_images
num_images: int
List
"""Sample images with many objects (cluttered scenes)."""
from
import
"coco"
"train"
for
in
"annotations"
if
len
10
if
len
break
return
def
extract_relevant_bboxes
image, question: str , answer: str
List
Tuple
"""
Extract bounding boxes for objects mentioned in question/answer.
In practice, use COCO annotations or manual annotation.
"""
from
import
from
import
as
True
eval
with
for
in
zip
0
"boxes"
0
"scores"
if
0.5
tuple
return
Step 2: Implement Vision-Grounded Answer Generation Train the model to produce both answers and bounding box evidence. First stage uses supervised fine-tuning with ground truth boxes:
import torch
import torch.nn as nn
from transformers import CLIPModel, AutoModel
class TreeVGRGroundedModel (nn.Module):
def __init__ (self, base_model_name="Qwen/Qwen2.5-VL-7B" ):
super ().__init__()
self .base_model = AutoModel.from_pretrained(base_model_name, trust_remote_code=True )
self .grounding_head = nn.Sequential(
nn.Linear(4096 , 2048 ),
nn.GELU(),
nn.Linear(2048 , 512 ),
nn.GELU(),
nn.Linear(512 , 4 )
)
self .answer_head = self .base_model.model.lm_head
def forward (self, image, question, question_type="general" ):
"""
Forward pass: encode image and question, predict boxes and answer.
"""
image_features = self .base_model.encode_image(image)
question_tokens = self .base_model.tokenize(question)
question_features = self .base_model.encode_text(question_tokens)
batch_size = image_features.shape[0 ]
h, w = image_features.shape[1 :3 ]
image_flat = image_features.reshape(batch_size, -1 , image_features.shape[-1 ])
attn_scores = torch.matmul(
question_features.unsqueeze(1 ),
image_flat.transpose(1 , 2 )
)
attn_weights = torch.softmax(attn_scores, dim=-1 )
attended_features = torch.matmul(
attn_weights,
image_flat
)
bbox_logits = self .grounding_head(attended_features.squeeze(1 ))
answer_logits = self .answer_head(
torch.cat([attended_features, question_features], dim=-1 )
)
return bbox_logits, answer_logits
def sft_training (model, train_dataset, num_epochs=2 ):
"""
Stage 1: Supervised fine-tuning with ground truth bounding boxes.
"""
optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5 )
loss_fn = nn.MSELoss()
for epoch in range (num_epochs):
for batch in train_dataset:
images = batch["image" ]
questions = batch["question" ]
answers = batch["answer" ]
bboxes = batch["bounding_boxes" ]
h, w = images.shape[2 :4 ]
bboxes_norm = bboxes / torch.tensor([w, h, w, h])
bbox_preds, answer_logits = model(images, questions)
bbox_loss = loss_fn(bbox_preds, bboxes_norm)
answer_loss = nn.CrossEntropyLoss()(
answer_logits.view(-1 , answer_logits.size(-1 )),
answers.view(-1 )
)
total_loss = 0.7 * bbox_loss + 0.3 * answer_loss
total_loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0 )
optimizer.step()
optimizer.zero_grad()
Step 3: Reinforcement Learning with Dual IoU Rewards Stage 2 refines predictions using RL with rewards for answer correctness and bounding box quality:
def compute_iou (pred_box: torch.Tensor, gt_box: torch.Tensor ) -> float :
"""Compute Intersection-over-Union between two boxes."""
x1_inter = max (pred_box[0 ].item(), gt_box[0 ].item())
y1_inter = max (pred_box[1 ].item(), gt_box[1 ].item())
x2_inter = min (pred_box[2 ].item(), gt_box[2 ].item())
y2_inter = min (pred_box[3 ].item(), gt_box[3 ].item())
inter_area = max (0 , x2_inter - x1_inter) * max (0 , y2_inter - y1_inter)
pred_area = (pred_box[2 ] - pred_box[0 ]) * (pred_box[3 ] - pred_box[1 ])
gt_area = (gt_box[2 ] - gt_box[0 ]) * (gt_box[3 ] - gt_box[1 ])
union_area = pred_area + gt_area - inter_area
if union_area == 0 :
return 0.0
return inter_area.item() / union_area.item()
def compute_grounding_reward (pred_boxes: List [torch.Tensor],
gt_boxes: List [torch.Tensor] ) -> Tuple [float , float ]:
"""
Compute dual IoU rewards: recall and precision.
Recall: fraction of ground truth boxes matched by predictions
Precision: fraction of predicted boxes that match ground truth
"""
recall_sum = 0
for gt_box in gt_boxes:
best_iou = max ([compute_iou(p, gt_box) for p in pred_boxes], default=0 )
recall_sum += 1 if best_iou > 0.5 else 0
recall = recall_sum / len (gt_boxes) if gt_boxes else 1.0
precision_sum = 0
for pred_box in pred_boxes:
best_iou = max ([compute_iou(pred_box, g) for g in gt_boxes], default=0 )
precision_sum += 1 if best_iou > 0.5 else 0
precision = precision_sum / len (pred_boxes) if pred_boxes else 1.0
return recall, precision
def rl_training (model, sft_model, train_dataset, num_epochs=1 ):
"""
Stage 2: Reinforcement learning with dual IoU rewards.
Optimizes answer accuracy + bounding box recall + precision.
"""
optimizer = torch.optim.AdamW(model.parameters(), lr=5e-6 )
for epoch in range (num_epochs):
for batch in train_dataset:
images = batch["image" ]
questions = batch["question" ]
answers = batch["answer" ]
gt_boxes = batch["bounding_boxes" ]
with torch.no_grad():
for _ in range (4 ):
bbox_preds, answer_logits = model(images, questions)
pred_answers = torch.argmax(answer_logits, dim=-1 )
h, w = images.shape[2 :4 ]
bbox_preds_denorm = bbox_preds * torch.tensor([w, h, w, h])
answer_correct = (pred_answers == answers).float ()
recall, precision = compute_grounding_reward(
bbox_preds_denorm, gt_boxes
)
reward = (
0.6 * answer_correct +
0.2 * recall +
0.2 * precision
)
loss = -reward.mean()
loss.backward()
optimizer.step()
optimizer.zero_grad()
Practical Guidance Component Recommended Value Notes Base Model Qwen2.5-VL-7B Strong vision-language foundation SFT Epochs 2 Enough to establish baseline RL Epochs 1 Additional refinement SFT Learning Rate 2e-5 Standard for instruction tuning RL Learning Rate 5e-6 Conservative for stability Answer Loss Weight 0.7 Prioritize correctness Bounding Box Loss Weight 0.3 Secondary emphasis during SFT Answer Reward Weight 0.6 Primary in RL Recall Reward Weight 0.2 Coverage of relevant objects Precision Reward Weight 0.2 Avoidance of false positives IoU Threshold 0.5 Standard object detection threshold TreeBench Samples 405 Diagnostic benchmark size Dense Object Scenes ~1000 base images Sample for dataset creation
VQA tasks emphasizing explainability and visual grounding
Scenarios where you need to debug model failures (boxes reveal reasoning)
Applications requiring traceable evidence for compliance
Fine-grained visual reasoning in cluttered scenes
When you want to prevent spurious correlation shortcuts
High-speed inference (bounding box prediction adds latency)
Simple factual QA where grounding is unnecessary
Scenes with very few objects (grounding is harder to interpret)
Memory-constrained deployment (additional grounding head)
Tasks where answer-only output is sufficient
Not normalizing bounding boxes to [0, 1] range before loss computation
IoU threshold too high (0.7+), making reward signal sparse
Forgetting to balance answer vs grounding rewards, leading to degenerate solutions
Not using ground truth boxes during SFT, limiting convergence
Treating recall and precision equally (adjust weights per domain)
Using arbitrary box coordinates instead of object centers
Not handling edge cases (images with 0 boxes, negative boxes)
Reference Li, S., Zhang, X., Chen, Y., & Wang, H. (2025). Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology. arXiv:2507.07999. https://arxiv.org/abs/2507.07999
