Evaluate text-to-video alignment through fine-grained semantic understanding via multi-agent question generation and knowledge-augmented answering. Generate 12,000 atomic yes/no questions from 2,000 prompts across 10 evaluation categories, achieving 58.47 correlation with human judgment.
Evaluate text-to-video alignment through fine-grained semantic understanding via multi-agent question generation and knowledge-augmented answering. Generate 12,000 atomic yes/no questions from 2,000 prompts across 10 evaluation categories, achieving 58.47 correlation with human judgment.
Core Concept
ETVA addresses a fundamental limitation in text-to-video (T2V) generation evaluation: existing metrics (CLIP score, FVD) fail to capture semantic alignment at fine-grained levels. Instead of crude similarity scores, ETVA simulates human annotation by decomposing natural language prompts into atomic yes/no questions covering semantics (objects, attributes, relationships), spatial-temporal properties (layout, motion), and physics (dynamics, physics simulation). A multi-agent system generates questions systematically, then a knowledge-augmented LLM answers them by reasoning through video content.
Architecture Overview
The ETVA evaluation framework consists of two integrated systems:
Question Generation (Multi-Agent): Element Extractor identifies semantic components (entities, attributes, relationships); Graph Builder constructs scene graphs representing dependencies; Graph Traverser systematically explores graphs to generate yes/no questions in dependency order
Question Answering (Knowledge-Augmented): An auxiliary LLM retrieves relevant commonsense knowledge (physics, spatial reasoning); a video LLM performs three-step analysis (video understanding, reflection with knowledge, conclusive answer)
The question generation pipeline extracts semantic elements from prompts and systematically generates atomic questions that probe different aspects of video content.
"""Extract entities, attributes, and relationships from text prompts."""
def
__init__
self, model_name="gpt-4"
self
self
def
extract_elements
self, prompt: str
List
"""Identify all semantic components in prompt."""
f"""
Extract semantic elements from this prompt:
"{prompt}"
Identify:
1. Entities (objects, people, animals)
2. Attributes (colors, sizes, materials, states)
3. Actions (verbs, motions)
4. Relationships (spatial, temporal)
Format each as: [type] [value] [dependencies]
"""
# Use LLM to extract elements
self
# Parse extraction results
for
in
'\n'
if
'['
if
len
3
1
']'
2
']'
0
type
set
return
class
SceneGraphBuilder
"""Construct hierarchical scene graph from semantic elements."""
def
__init__
self
self
def
build_graph
self, elements: List[SemanticElement]
Dict
"""Create directed acyclic graph of semantic dependencies."""
# Initialize nodes
for
in
self
'type'
type
'dependencies'
'dependents'
# Identify dependencies between elements
# (e.g., "red cube" depends on "red" and "cube")
f"""
Given these elements: {[e.value for e in elements]}
Identify dependencies (which elements are prerequisites for others).
Format: [dependent] <- [prerequisite]
"""
self
# Parse and update graph
for
in
'\n'
if
'<-'
in
'<-'
0
1
if
in
self
and
in
self
self
'dependencies'
self
'dependents'
return
self
class
GraphTraverser
"""Systematically traverse scene graph to generate atomic questions."""
def
__init__
self, scene_graph: Dict
self
self
set
def
generate_questions
self
List
str
"""Generate yes/no questions following dependency order."""
The QA stage uses an auxiliary LLM to retrieve commonsense knowledge (especially critical for physics questions), then has a video LLM perform multi-step reasoning through the video.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
classKnowledgeAugmentedQA:
"""Answer yes/no questions about video with external knowledge."""def__init__(
self,
knowledge_model_name="Qwen2.5-72B",
video_model_name="Qwen2-VL-72B"):
self.knowledge_model = AutoModelForCausalLM.from_pretrained(
knowledge_model_name
)
self.video_model = AutoModelForCausalLM.from_pretrained(
video_model_name
)
self.knowledge_tokenizer = AutoTokenizer.from_pretrained(
knowledge_model_name
)
self.video_tokenizer = AutoTokenizer.from_pretrained(
video_model_name
)
self.knowledge_cache = {}
defretrieve_knowledge(self, question: str) -> str:
"""Retrieve commonsense knowledge for context."""# Check cache firstif question inself.knowledge_cache:
returnself.knowledge_cache[question]
# Retrieve knowledge for specific domains
knowledge_prompt = f"""
Question: {question}
Provide relevant background knowledge or physical principles that help answer this question.
Keep response concise (2-3 sentences).
"""
knowledge = self._call_knowledge_model(knowledge_prompt)
self.knowledge_cache[question] = knowledge
return knowledge
defanswer_question_multistage(
self, video, question: str) -> Dict[str, any]:
"""Three-stage reasoning: understand, reflect, answer."""# Stage 1: Video Understanding
understanding_prompt = f"""
Watch this video and describe what you see:
[VIDEO_PLACEHOLDER]
Focus on: objects, actions, spatial layout, temporal progression.
Keep description under 100 words.
"""
video_understanding = self._call_video_model(
video, understanding_prompt
)
# Stage 2: Contextual Reflection with Knowledge
knowledge = self.retrieve_knowledge(question)
reflection_prompt = f"""
Video content: {video_understanding}
Background knowledge: {knowledge}
Question: {question}
What aspects of the video are relevant to answering this question?
"""
reflection = self._call_video_model(video, reflection_prompt)
# Stage 3: Conclusive Answer
answer_prompt = f"""
Video: {video_understanding}
Knowledge: {knowledge}
Reflection: {reflection}
Question: {question}
Answer YES or NO. Provide brief justification (1-2 sentences).
"""
answer_result = self._call_video_model(video, answer_prompt)
# Parse answer
answer_lower = answer_result.lower()
is_yes = "yes"in answer_lower or answer_lower.startswith("yes")
return {
'question': question,
'answer': "YES"if is_yes else"NO",
'confidence': self._extract_confidence(answer_result),
'reasoning': answer_result,
'video_understanding': video_understanding,
'knowledge': knowledge,
'reflection': reflection
}
def_call_knowledge_model(self, prompt: str) -> str:
"""Query knowledge model."""
inputs = self.knowledge_tokenizer(prompt, return_tensors="pt")
outputs = self.knowledge_model.generate(
**inputs, max_length=150, temperature=0.7
)
returnself.knowledge_tokenizer.decode(
outputs[0], skip_special_tokens=True
)
def_call_video_model(
self, video: torch.Tensor, prompt: str) -> str:
"""Query video understanding model."""# In practice: properly encode video frames
inputs = self.video_tokenizer(prompt, return_tensors="pt")
outputs = self.video_model.generate(
**inputs, max_length=200, temperature=0.7
)
returnself.video_tokenizer.decode(
outputs[0], skip_special_tokens=True
)
def_extract_confidence(self, response: str) -> float:
"""Extract confidence score from model response."""# Simple heuristic: count certainty markers
certainty_markers = ["definitely", "clearly", "certainly", "no doubt"]
uncertainty_markers = ["might", "seems", "appears", "uncertain"]
certainty_count = sum(
1for marker in certainty_markers
if marker in response.lower()
)
uncertainty_count = sum(
1for marker in uncertainty_markers
if marker in response.lower()
)
confidence = (certainty_count - uncertainty_count) / max(
certainty_count + uncertainty_count, 1
)
returnmax(0.0, min(1.0, 0.5 + confidence * 0.5))
defevaluate_video_qa(
video, generated_video, questions: List[str]
) -> Dict:
"""Evaluate how well generated video answers atomic questions."""
qa_system = KnowledgeAugmentedQA()
results = {
'question_answers': [],
'accuracy': 0.0,
'category_performance': {}
}
correct_count = 0for question in questions:
# Get answer for both reference and generated video
ref_answer = qa_system.answer_question_multistage(
video, question
)
gen_answer = qa_system.answer_question_multistage(
generated_video, question
)
# Check alignment
aligned = (ref_answer['answer'] == gen_answer['answer'])
correct_count += int(aligned)
results['question_answers'].append({
'question': question,
'reference_answer': ref_answer['answer'],
'generated_answer': gen_answer['answer'],
'aligned': aligned
})
results['accuracy'] = correct_count / len(questions)
return results
ETVABench Benchmark Construction and Evaluation
Build a comprehensive benchmark with diverse prompts covering 10 evaluation categories, then evaluate T2V models systematically.
from enum import Enum
classEvaluationCategory(Enum):
"""10 evaluation categories for text-to-video assessment."""
EXISTENCE = "existence"# Does object exist?
ACTION = "action"# Is action performed?
MATERIAL = "material"# Object material properties?
SPATIAL = "spatial"# Spatial arrangement?
NUMBER = "number"# Quantity of objects?
SHAPE = "shape"# Shape properties?
COLOR = "color"# Color attributes?
CAMERA = "camera"# Camera control/movement?
PHYSICS = "physics"# Physics simulation?
OTHER = "other"# Other semantic propertiesclassETVABench:
"""Text-to-video alignment benchmark with 2000 prompts."""def__init__(self):
self.prompts = []
self.questions_by_prompt = {}
self.category_distribution = {}
defconstruct_benchmark(self, num_prompts: int = 2000):
"""Build benchmark with diverse prompts across categories."""# Load or generate diverse prompts
prompts_per_category = num_prompts // len(EvaluationCategory)
for category in EvaluationCategory:
category_prompts = self._generate_prompts_for_category(
category, prompts_per_category
)
self.prompts.extend(category_prompts)
# Generate questions for each promptfor prompt in category_prompts:
questions = generate_questions_for_prompt(prompt)
self.questions_by_prompt[prompt] = questions
self.category_distribution[category.value] = len(
category_prompts
)
def_generate_prompts_for_category(
self, category: EvaluationCategory, count: int) -> List[str]:
"""Generate diverse prompts for evaluation category."""
prompts = []
if category == EvaluationCategory.EXISTENCE:
templates = [
"A {object} in a {setting}",
"{object} doing {action}",
"{object} with {attribute} {property}",
]
elif category == EvaluationCategory.ACTION:
templates = [
"{object} {action} in {setting}",
"{object} slowly {action}",
"{object} quickly {action}",
]
elif category == EvaluationCategory.PHYSICS:
templates = [
"{object} falling under gravity in {setting}",
"{object} floating in {environment}",
"{object} bouncing on {surface}",
]
# Expand templates with variationsfor template in templates[:count]:
prompt = template.replace(
"{object}", "cat"
).replace(
"{action}", "running"
).replace(
"{setting}", "a park"
)
prompts.append(prompt)
return prompts
defevaluate_t2v_models(
t2v_models: Dict[str, any],
benchmark: ETVABench
) -> Dict[str, Dict]:
"""Evaluate multiple T2V models on ETVABench."""
evaluation_results = {}
for model_name, model in t2v_models.items():
print(f"\nEvaluating {model_name}...")
model_results = {
'overall_accuracy': 0.0,
'category_accuracy': {},
'temporal_accuracy': 0.0,
'physics_accuracy': 0.0
}
category_accuracies = {}
all_accuracies = []
for prompt in benchmark.prompts:
# Generate video
generated_video = model.generate(prompt)
# Get questions for this prompt
questions = benchmark.questions_by_prompt[prompt]
# Evaluate alignment
qa_results = evaluate_video_qa(
None, generated_video, questions
)
all_accuracies.append(qa_results['accuracy'])
# Track by category
category = categorize_prompt(prompt)
if category notin category_accuracies:
category_accuracies[category] = []
category_accuracies[category].append(qa_results['accuracy'])
# Aggregate results
model_results['overall_accuracy'] = (
sum(all_accuracies) / len(all_accuracies)
)
for category, accuracies in category_accuracies.items():
model_results['category_accuracy'][category] = (
sum(accuracies) / len(accuracies)
)
evaluation_results[model_name] = model_results
return evaluation_results
defcategorize_prompt(prompt: str) -> str:
"""Categorize prompt into evaluation category."""
prompt_lower = prompt.lower()
ifany(word in prompt_lower for word in ["exist", "contain", "has"]):
return"existence"elifany(word in prompt_lower for word in ["falling", "bouncing", "gravity", "float"]):
return"physics"elifany(word in prompt_lower for word in ["camera", "pan", "zoom", "move"]):
return"camera"else:
return"other"
Practical Guidance
When to use ETVA:
You're developing or evaluating T2V models and need fine-grained semantic alignment assessment
You want to identify specific weaknesses (e.g., physics simulation, camera control) in generated videos
You need evaluation that correlates well with human judgment (Spearman's ρ = 58.47 vs. VideoScore's 31.0)
You're building a benchmark for reproducible T2V evaluation across multiple models
When NOT to use:
You need real-time evaluation (multi-agent QA + video understanding is computationally expensive)
Your videos are extremely short (< 2 seconds) where atomic question answering is unreliable
You need to evaluate non-semantic aspects (technical quality, compression artifacts)
Budget is extremely limited (requires multiple LLM API calls per video)
Hyperparameter and design choices:
Number of atomic questions: 6 questions per prompt typical; increase to 10 for complex scenes, reduce to 3 for simple objects
Knowledge model: Qwen2.5-72B recommended for physics/reasoning; GPT-4 acceptable if budget allows
Video LLM: Qwen2-VL-72B for open-source; GPT-4V or Gemini for closed-source (may improve accuracy)
Question categories: 10 core categories defined; customize based on evaluation priorities
Multi-stage reasoning steps: 3 stages (understand, reflect, answer) optimal; can reduce to 2 for speed
Common pitfalls:
Insufficient knowledge augmentation: Skipping the knowledge retrieval stage causes physics questions to fail; always retrieve domain-specific knowledge
Weak question generation: Vanilla LLM prompting generates redundant questions; use multi-agent graph traversal for systematic coverage
Evaluation category imbalance: If physics represents only 10% of prompts, physics limitations aren't well-detected; balance categories for comprehensive assessment
Video understanding failures: If video LLM struggles to understand generated video content, all downstream QA fails; validate video LLM separately
Yes/no answer ambiguity: Models may give nuanced answers; parse answers strictly for binary yes/no to avoid ambiguity
Benchmark: ETVABench with 2,000 prompts generating 12,000 atomic questions across 10 categories
Evaluation metrics: Spearman's ρ correlation with human judgment (58.47 overall), ablation studies showing 34.1% improvement from multi-agent QA, 21.93% from knowledge augmentation, 6.5-11.2% from multi-stage reasoning
Key findings: All 15 evaluated T2V models struggle with temporal dynamics (physics max 0.600, camera max 0.474); static attribute generation is stronger than spatiotemporal reasoning
Evaluation dataset: Manually deconstructed CoT steps (FlowVerse-CoT-E) more robust than automatic step extraction
Related work: CLIP-based metrics, FVD (Fréchet Video Distance), VideoScore baseline evaluation methods
