菜单
Evaluate text-to-video alignment through fine-grained semantic understanding via multi-agent question generation and knowledge-augmented answering. Generate 12,000 atomic yes/no questions from 2,000 prompts across 10 evaluation categories, achieving 58.47 correlation with human judgment.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/ADu2021/skillXiv --skill text-video-alignment-evaluation命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
基于 SOC 职业分类
正在显示 SKILL.md
| name | text-video-alignment-evaluation |
| title | ETVA: Evaluation of Text-to-Video Alignment via Fine-grained Question Generation and Answering |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2503.16867 |
| keywords | ["Text-to-Video Generation","Evaluation Metrics","Question Generation","Video Understanding","Benchmark"] |
| description | Evaluate text-to-video alignment through fine-grained semantic understanding via multi-agent question generation and knowledge-augmented answering. Generate 12,000 atomic yes/no questions from 2,000 prompts across 10 evaluation categories, achieving 58.47 correlation with human judgment. |
ETVA addresses a fundamental limitation in text-to-video (T2V) generation evaluation: existing metrics (CLIP score, FVD) fail to capture semantic alignment at fine-grained levels. Instead of crude similarity scores, ETVA simulates human annotation by decomposing natural language prompts into atomic yes/no questions covering semantics (objects, attributes, relationships), spatial-temporal properties (layout, motion), and physics (dynamics, physics simulation). A multi-agent system generates questions systematically, then a knowledge-augmented LLM answers them by reasoning through video content.
The ETVA evaluation framework consists of two integrated systems:
The question generation pipeline extracts semantic elements from prompts and systematically generates atomic questions that probe different aspects of video content.
from dataclasses import dataclass
from typing import List, Dict, Set
import json
from transformers import AutoTokenizer, AutoModelForCausalLM
@dataclass
class SemanticElement:
"""Represents a semantic element extracted from prompt."""
type: str # "entity", "attribute", "relationship", "action"
value: str
dependencies: [] =
:
():
.model_name = model_name
.tokenizer = AutoTokenizer.from_pretrained(model_name)
() -> [SemanticElement]:
extraction_prompt =
elements = []
extraction_result = ._call_llm(extraction_prompt)
line extraction_result.split():
line.strip():
parts = line.split()
(parts) >= :
element_type = parts[].strip().strip()
value = parts[].split()[].strip()
element = SemanticElement(
=element_type,
value=value,
dependencies=()
)
elements.append(element)
elements
:
():
.graph = {}
() -> :
elem elements:
.graph[elem.value] = {
: elem.,
: [],
: []
}
dependency_prompt =
dependency_result = ._call_llm(dependency_prompt)
line dependency_result.split():
line:
parts = line.split()
dependent = parts[].strip()
prerequisite = parts[].strip()
dependent .graph prerequisite .graph:
.graph[dependent][].append(prerequisite)
.graph[prerequisite][].append(dependent)
.graph
:
():
.scene_graph = scene_graph
.visited = ()
() -> []:
questions = []
sorted_elements = ._topological_sort()
element sorted_elements:
node = .scene_graph[element]
node[] == :
questions.append(
)
node[] == :
questions.append(
)
dependent node[]:
questions.append(
)
node[] == :
questions.append(
)
node[] == :
questions.append(
)
questions
() -> []:
visited = ()
sorted_list = []
():
node visited:
visited.add(node)
dep .scene_graph[node][]:
dfs(dep)
sorted_list.append(node)
node .scene_graph.keys():
dfs(node)
sorted_list
() -> []:
extractor = ElementExtractor()
elements = extractor.extract_elements(prompt)
graph_builder = SceneGraphBuilder()
scene_graph = graph_builder.build_graph(elements)
traverser = GraphTraverser(scene_graph)
questions = traverser.generate_questions()
questions
The QA stage uses an auxiliary LLM to retrieve commonsense knowledge (especially critical for physics questions), then has a video LLM perform multi-step reasoning through the video.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
class KnowledgeAugmentedQA:
"""Answer yes/no questions about video with external knowledge."""
def __init__(
self,
knowledge_model_name="Qwen2.5-72B",
video_model_name="Qwen2-VL-72B"
):
self.knowledge_model = AutoModelForCausalLM.from_pretrained(
knowledge_model_name
)
self.video_model = AutoModelForCausalLM.from_pretrained(
video_model_name
)
self.knowledge_tokenizer = AutoTokenizer.from_pretrained(
knowledge_model_name
)
self.video_tokenizer = AutoTokenizer.from_pretrained(
video_model_name
)
self.knowledge_cache = {}
def retrieve_knowledge(self, question: str) -> str:
"""Retrieve commonsense knowledge for context."""
# Check cache first
if question in self.knowledge_cache:
return self.knowledge_cache[question]
# Retrieve knowledge for specific domains
knowledge_prompt = f"""
Question: {question}
Provide relevant background knowledge or physical principles that help answer this question.
Keep response concise (2-3 sentences).
"""
knowledge = self._call_knowledge_model(knowledge_prompt)
self.knowledge_cache[question] = knowledge
return knowledge
def answer_question_multistage(
self, video, question: str
) -> Dict[str, any]:
"""Three-stage reasoning: understand, reflect, answer."""
# Stage 1: Video Understanding
understanding_prompt = f"""
Watch this video and describe what you see:
[VIDEO_PLACEHOLDER]
Focus on: objects, actions, spatial layout, temporal progression.
Keep description under 100 words.
"""
video_understanding = self._call_video_model(
video, understanding_prompt
)
# Stage 2: Contextual Reflection with Knowledge
knowledge = self.retrieve_knowledge(question)
reflection_prompt = f"""
Video content: {video_understanding}
Background knowledge: {knowledge}
Question: {question}
What aspects of the video are relevant to answering this question?
"""
reflection = self._call_video_model(video, reflection_prompt)
# Stage 3: Conclusive Answer
answer_prompt = f"""
Video: {video_understanding}
Knowledge: {knowledge}
Reflection: {reflection}
Question: {question}
Answer YES or NO. Provide brief justification (1-2 sentences).
"""
answer_result = self._call_video_model(video, answer_prompt)
# Parse answer
answer_lower = answer_result.lower()
is_yes = "yes" in answer_lower or answer_lower.startswith("yes")
return {
'question': question,
'answer': "YES" if is_yes else "NO",
'confidence': self._extract_confidence(answer_result),
'reasoning': answer_result,
'video_understanding': video_understanding,
'knowledge': knowledge,
'reflection': reflection
}
def _call_knowledge_model(self, prompt: str) -> str:
"""Query knowledge model."""
inputs = self.knowledge_tokenizer(prompt, return_tensors="pt")
outputs = self.knowledge_model.generate(
**inputs, max_length=150, temperature=0.7
)
return self.knowledge_tokenizer.decode(
outputs[0], skip_special_tokens=True
)
def _call_video_model(
self, video: torch.Tensor, prompt: str
) -> str:
"""Query video understanding model."""
# In practice: properly encode video frames
inputs = self.video_tokenizer(prompt, return_tensors="pt")
outputs = self.video_model.generate(
**inputs, max_length=200, temperature=0.7
)
return self.video_tokenizer.decode(
outputs[0], skip_special_tokens=True
)
def _extract_confidence(self, response: str) -> float:
"""Extract confidence score from model response."""
# Simple heuristic: count certainty markers
certainty_markers = ["definitely", "clearly", "certainly", "no doubt"]
uncertainty_markers = ["might", "seems", "appears", "uncertain"]
certainty_count = sum(
1 for marker in certainty_markers
if marker in response.lower()
)
uncertainty_count = sum(
1 for marker in uncertainty_markers
if marker in response.lower()
)
confidence = (certainty_count - uncertainty_count) / max(
certainty_count + uncertainty_count, 1
)
return max(0.0, min(1.0, 0.5 + confidence * 0.5))
def evaluate_video_qa(
video, generated_video, questions: List[str]
) -> Dict:
"""Evaluate how well generated video answers atomic questions."""
qa_system = KnowledgeAugmentedQA()
results = {
'question_answers': [],
'accuracy': 0.0,
'category_performance': {}
}
correct_count = 0
for question in questions:
# Get answer for both reference and generated video
ref_answer = qa_system.answer_question_multistage(
video, question
)
gen_answer = qa_system.answer_question_multistage(
generated_video, question
)
# Check alignment
aligned = (ref_answer['answer'] == gen_answer['answer'])
correct_count += int(aligned)
results['question_answers'].append({
'question': question,
'reference_answer': ref_answer['answer'],
'generated_answer': gen_answer['answer'],
'aligned': aligned
})
results['accuracy'] = correct_count / len(questions)
return results
Build a comprehensive benchmark with diverse prompts covering 10 evaluation categories, then evaluate T2V models systematically.
from enum import Enum
class EvaluationCategory(Enum):
"""10 evaluation categories for text-to-video assessment."""
EXISTENCE = "existence" # Does object exist?
ACTION = "action" # Is action performed?
MATERIAL = "material" # Object material properties?
SPATIAL = "spatial" # Spatial arrangement?
NUMBER = "number" # Quantity of objects?
SHAPE = "shape" # Shape properties?
COLOR = "color" # Color attributes?
CAMERA = "camera" # Camera control/movement?
PHYSICS = "physics" # Physics simulation?
OTHER = "other" # Other semantic properties
class ETVABench:
"""Text-to-video alignment benchmark with 2000 prompts."""
def __init__(self):
self.prompts = []
self.questions_by_prompt = {}
self.category_distribution = {}
def construct_benchmark(self, num_prompts: int = 2000):
"""Build benchmark with diverse prompts across categories."""
# Load or generate diverse prompts
prompts_per_category = num_prompts // len(EvaluationCategory)
for category in EvaluationCategory:
category_prompts = self._generate_prompts_for_category(
category, prompts_per_category
)
self.prompts.extend(category_prompts)
# Generate questions for each prompt
for prompt in category_prompts:
questions = generate_questions_for_prompt(prompt)
self.questions_by_prompt[prompt] = questions
self.category_distribution[category.value] = len(
category_prompts
)
def _generate_prompts_for_category(
self, category: EvaluationCategory, count: int
) -> List[str]:
"""Generate diverse prompts for evaluation category."""
prompts = []
if category == EvaluationCategory.EXISTENCE:
templates = [
"A {object} in a {setting}",
"{object} doing {action}",
"{object} with {attribute} {property}",
]
elif category == EvaluationCategory.ACTION:
templates = [
"{object} {action} in {setting}",
"{object} slowly {action}",
"{object} quickly {action}",
]
elif category == EvaluationCategory.PHYSICS:
templates = [
"{object} falling under gravity in {setting}",
"{object} floating in {environment}",
"{object} bouncing on {surface}",
]
# Expand templates with variations
for template in templates[:count]:
prompt = template.replace(
"{object}", "cat"
).replace(
"{action}", "running"
).replace(
"{setting}", "a park"
)
prompts.append(prompt)
return prompts
def evaluate_t2v_models(
t2v_models: Dict[str, any],
benchmark: ETVABench
) -> Dict[str, Dict]:
"""Evaluate multiple T2V models on ETVABench."""
evaluation_results = {}
for model_name, model in t2v_models.items():
print(f"\nEvaluating {model_name}...")
model_results = {
'overall_accuracy': 0.0,
'category_accuracy': {},
'temporal_accuracy': 0.0,
'physics_accuracy': 0.0
}
category_accuracies = {}
all_accuracies = []
for prompt in benchmark.prompts:
# Generate video
generated_video = model.generate(prompt)
# Get questions for this prompt
questions = benchmark.questions_by_prompt[prompt]
# Evaluate alignment
qa_results = evaluate_video_qa(
None, generated_video, questions
)
all_accuracies.append(qa_results['accuracy'])
# Track by category
category = categorize_prompt(prompt)
if category not in category_accuracies:
category_accuracies[category] = []
category_accuracies[category].append(qa_results['accuracy'])
# Aggregate results
model_results['overall_accuracy'] = (
sum(all_accuracies) / len(all_accuracies)
)
for category, accuracies in category_accuracies.items():
model_results['category_accuracy'][category] = (
sum(accuracies) / len(accuracies)
)
evaluation_results[model_name] = model_results
return evaluation_results
def categorize_prompt(prompt: str) -> str:
"""Categorize prompt into evaluation category."""
prompt_lower = prompt.lower()
if any(word in prompt_lower for word in ["exist", "contain", "has"]):
return "existence"
elif any(word in prompt_lower for word in ["falling", "bouncing", "gravity", "float"]):
return "physics"
elif any(word in prompt_lower for word in ["camera", "pan", "zoom", "move"]):
return "camera"
else:
return "other"
When to use ETVA:
When NOT to use:
Hyperparameter and design choices:
Common pitfalls: