Skip to main content ホーム クリエイター adu2021 skillxiv treevgr-visual-grounded-reasoning-evidence
treevgr-visual-grounded-reasoning-evidence Train vision-language models to produce visually grounded reasoning by enforcing traceable evidence via bounding box localization, using a novel benchmark (TreeBench) and RL-based training with dual IoU rewards for both recall and precision.
インストールへ移動 Skills Marketplace コミュニティが作成したAIスキルを発見・探索
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/ADu2021/skillXiv --skill treevgr-visual-grounded-reasoning-evidenceコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
Zipをダウンロード ダウンロード中... このリポジトリの他の Skills meaningful-kebab-case-name Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
action-quantization-behavior-cloning Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
adaptive-lora-personalized-ranks Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
name treevgr-visual-grounded-reasoning-evidence title Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2507.07999 keywords ["Visual Reasoning","Evidence Grounding","Reinforcement Learning","Object Localization","VQA"] description Train vision-language models to produce visually grounded reasoning by enforcing traceable evidence via bounding box localization, using a novel benchmark (TreeBench) and RL-based training with dual IoU rewards for both recall and precision.
TreeVGR: Visual Grounded Reasoning with Traceable Evidence
Standard vision-language models answer questions but rarely show their visual reasoning. TreeVGR introduces a training paradigm and benchmark that demand evidence: the model must identify and localize specific regions supporting its answers. TreeBench is a diagnostic benchmark of 405 carefully curated VQA pairs emphasizing fine perception in cluttered scenes. TreeVGR trains models to produce bounding boxes alongside answers, using reinforcement learning with dual IoU metrics that reward both recall (did you find all relevant objects?) and precision (did you avoid false positives?).
This approach transforms VQA from a classification task into a grounded reasoning task, improving explainability and robustness.
Core Concept
Visual reasoning without visual evidence is brittle. Models can guess correctly by exploiting spurious correlations. TreeVGR enforces visual grounding: to answer a question, the model must identify which objects or regions support the answer. The training reward combines three signals: (1) answer correctness (did you get it right?), (2) bounding box recall (did you localize all relevant objects?), (3) bounding box precision (did you avoid false positives?). This multi-faceted reward encourages genuine visual understanding rather than shortcut learning.
The benchmark carefully samples cluttered scenes (dense objects) and emphasizes second-order reasoning (reasoning about attributes, spatial relations, transformations) beyond simple localization.
Architecture Overview
Base Model : Qwen2.5-VL-7B vision-language modelQuestion-Guided Vision : Extract features conditioned on query, focus on relevant regionsGrounding Head : Predict bounding boxes for evidence regionsAnswer Head : Generate answer text conditioned on grounded featuresTreeBench : 405 VQA pairs with bounding box annotations in cluttered scenesDual IoU Rewards : Separate metrics for recall and precision in localization
Implementation
Step 1: Create TreeBench-Style Evaluation Dataset
Curate a dataset emphasizing fine visual perception and evidence-based reasoning. Select dense object scenes and annotate questions across two categories:
import json
import torch
from typing import List , Dict , Tuple
from dataclasses import
@dataclass
class
TreeBenchExample
str
str
str
List
Tuple
int
int
int
int
str
str
def
create_treebench_dataset
num_examples=405 , num_images=1000
"""
Create TreeBench: VQA dataset emphasizing visual perception
in dense object scenes with bounding box annotations.
"""
for
in
enumerate
"color"
f"What color is the {random_object(image)} ?"
"material"
f"What material is the {random_object(image)} ?"
"attribute"
f"Describe the {random_object(image)} ."
"spatial"
f"Where is the {random_object(image)} located?"
"count"
f"How many objects are {random_property(image)} ?"
"perspective"
f"What would you see from {random_direction(image)} ?"
"containment"
f"Is the {random_object(image)} inside or outside?"
"transformation"
f"If we rotate the {random_object(image)} 90°, where would it be?"
"relationship"
f"What is the relationship between {random_object(image)} and {random_object(image)} ?"
"inference"
f"Based on the scene, what might happen next?"
for
in
"perception"
if
in
0
for
in
else
"reasoning"
return
def
sample_dense_object_images
num_images: int
List
"""Sample images with many objects (cluttered scenes)."""
from
import
"coco"
"train"
for
in
"annotations"
if
len
10
if
len
break
return
def
extract_relevant_bboxes
image, question: str , answer: str
List
Tuple
"""
Extract bounding boxes for objects mentioned in question/answer.
In practice, use COCO annotations or manual annotation.
"""
from
import
from
import
as
True
eval
with
for
in
zip
0
"boxes"
0
"scores"
if
0.5
tuple
return
Step 2: Implement Vision-Grounded Answer Generation Train the model to produce both answers and bounding box evidence. First stage uses supervised fine-tuning with ground truth boxes:
import torch
import torch.nn as nn
from transformers import CLIPModel, AutoModel
class TreeVGRGroundedModel (nn.Module):
def __init__ (self, base_model_name="Qwen/Qwen2.5-VL-7B" ):
super ().__init__()
self .base_model = AutoModel.from_pretrained(base_model_name, trust_remote_code=True )
self .grounding_head = nn.Sequential(
nn.Linear(4096 , 2048 ),
nn.GELU(),
nn.Linear(2048 , 512 ),
nn.GELU(),
nn.Linear(512 , 4 )
)
self .answer_head = self .base_model.model.lm_head
def forward (self, image, question, question_type="general" ):
"""
Forward pass: encode image and question, predict boxes and answer.
"""
image_features = self .base_model.encode_image(image)
question_tokens = self .base_model.tokenize(question)
question_features = self .base_model.encode_text(question_tokens)
batch_size = image_features.shape[0 ]
h, w = image_features.shape[1 :3 ]
image_flat = image_features.reshape(batch_size, -1 , image_features.shape[-1 ])
attn_scores = torch.matmul(
question_features.unsqueeze(1 ),
image_flat.transpose(1 , 2 )
)
attn_weights = torch.softmax(attn_scores, dim=-1 )
attended_features = torch.matmul(
attn_weights,
image_flat
)
bbox_logits = self .grounding_head(attended_features.squeeze(1 ))
answer_logits = self .answer_head(
torch.cat([attended_features, question_features], dim=-1 )
)
return bbox_logits, answer_logits
def sft_training (model, train_dataset, num_epochs=2 ):
"""
Stage 1: Supervised fine-tuning with ground truth bounding boxes.
"""
optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5 )
loss_fn = nn.MSELoss()
for epoch in range (num_epochs):
for batch in train_dataset:
images = batch["image" ]
questions = batch["question" ]
answers = batch["answer" ]
bboxes = batch["bounding_boxes" ]
h, w = images.shape[2 :4 ]
bboxes_norm = bboxes / torch.tensor([w, h, w, h])
bbox_preds, answer_logits = model(images, questions)
bbox_loss = loss_fn(bbox_preds, bboxes_norm)
answer_loss = nn.CrossEntropyLoss()(
answer_logits.view(-1 , answer_logits.size(-1 )),
answers.view(-1 )
)
total_loss = 0.7 * bbox_loss + 0.3 * answer_loss
total_loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0 )
optimizer.step()
optimizer.zero_grad()
Step 3: Reinforcement Learning with Dual IoU Rewards Stage 2 refines predictions using RL with rewards for answer correctness and bounding box quality:
def compute_iou (pred_box: torch.Tensor, gt_box: torch.Tensor ) -> float :
"""Compute Intersection-over-Union between two boxes."""
x1_inter = max (pred_box[0 ].item(), gt_box[0 ].item())
y1_inter = max (pred_box[1 ].item(), gt_box[1 ].item())
x2_inter = min (pred_box[2 ].item(), gt_box[2 ].item())
y2_inter = min (pred_box[3 ].item(), gt_box[3 ].item())
inter_area = max (0 , x2_inter - x1_inter) * max (0 , y2_inter - y1_inter)
pred_area = (pred_box[2 ] - pred_box[0 ]) * (pred_box[3 ] - pred_box[1 ])
gt_area = (gt_box[2 ] - gt_box[0 ]) * (gt_box[3 ] - gt_box[1 ])
union_area = pred_area + gt_area - inter_area
if union_area == 0 :
return 0.0
return inter_area.item() / union_area.item()
def compute_grounding_reward (pred_boxes: List [torch.Tensor],
gt_boxes: List [torch.Tensor] ) -> Tuple [float , float ]:
"""
Compute dual IoU rewards: recall and precision.
Recall: fraction of ground truth boxes matched by predictions
Precision: fraction of predicted boxes that match ground truth
"""
recall_sum = 0
for gt_box in gt_boxes:
best_iou = max ([compute_iou(p, gt_box) for p in pred_boxes], default=0 )
recall_sum += 1 if best_iou > 0.5 else 0
recall = recall_sum / len (gt_boxes) if gt_boxes else 1.0
precision_sum = 0
for pred_box in pred_boxes:
best_iou = max ([compute_iou(pred_box, g) for g in gt_boxes], default=0 )
precision_sum += 1 if best_iou > 0.5 else 0
precision = precision_sum / len (pred_boxes) if pred_boxes else 1.0
return recall, precision
def rl_training (model, sft_model, train_dataset, num_epochs=1 ):
"""
Stage 2: Reinforcement learning with dual IoU rewards.
Optimizes answer accuracy + bounding box recall + precision.
"""
optimizer = torch.optim.AdamW(model.parameters(), lr=5e-6 )
for epoch in range (num_epochs):
for batch in train_dataset:
images = batch["image" ]
questions = batch["question" ]
answers = batch["answer" ]
gt_boxes = batch["bounding_boxes" ]
with torch.no_grad():
for _ in range (4 ):
bbox_preds, answer_logits = model(images, questions)
pred_answers = torch.argmax(answer_logits, dim=-1 )
h, w = images.shape[2 :4 ]
bbox_preds_denorm = bbox_preds * torch.tensor([w, h, w, h])
answer_correct = (pred_answers == answers).float ()
recall, precision = compute_grounding_reward(
bbox_preds_denorm, gt_boxes
)
reward = (
0.6 * answer_correct +
0.2 * recall +
0.2 * precision
)
loss = -reward.mean()
loss.backward()
optimizer.step()
optimizer.zero_grad()
Practical Guidance Component Recommended Value Notes Base Model Qwen2.5-VL-7B Strong vision-language foundation SFT Epochs 2 Enough to establish baseline RL Epochs 1 Additional refinement SFT Learning Rate 2e-5 Standard for instruction tuning RL Learning Rate 5e-6 Conservative for stability Answer Loss Weight 0.7 Prioritize correctness Bounding Box Loss Weight 0.3 Secondary emphasis during SFT Answer Reward Weight 0.6 Primary in RL Recall Reward Weight 0.2 Coverage of relevant objects Precision Reward Weight 0.2 Avoidance of false positives IoU Threshold 0.5 Standard object detection threshold TreeBench Samples 405 Diagnostic benchmark size Dense Object Scenes ~1000 base images Sample for dataset creation
VQA tasks emphasizing explainability and visual grounding
Scenarios where you need to debug model failures (boxes reveal reasoning)
Applications requiring traceable evidence for compliance
Fine-grained visual reasoning in cluttered scenes
When you want to prevent spurious correlation shortcuts
High-speed inference (bounding box prediction adds latency)
Simple factual QA where grounding is unnecessary
Scenes with very few objects (grounding is harder to interpret)
Memory-constrained deployment (additional grounding head)
Tasks where answer-only output is sufficient
Not normalizing bounding boxes to [0, 1] range before loss computation
IoU threshold too high (0.7+), making reward signal sparse
Forgetting to balance answer vs grounding rewards, leading to degenerate solutions
Not using ground truth boxes during SFT, limiting convergence
Treating recall and precision equally (adjust weights per domain)
Using arbitrary box coordinates instead of object centers
Not handling edge cases (images with 0 boxes, negative boxes)
Reference Li, S., Zhang, X., Chen, Y., & Wang, H. (2025). Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology. arXiv:2507.07999. https://arxiv.org/abs/2507.07999
