Skip to main content Accueil Créateurs adu2021 skillxiv mact-document-understanding
mact-document-understanding Decomposes document processing into specialized agents (planning, execution, judgment, answer) with agent-wise adaptive test-time scaling. Achieves 9.9-11.5% performance gain with smaller models while maintaining reasoning.
Aller à l'installation Skills Marketplace Découvrez et explorez les compétences IA créées par la communauté.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Afficher les détails du prompt Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
npx skills add https://github.com/ADu2021/skillXiv --skill mact-document-understandingLa commande reste sur une seule ligne. Faites défiler horizontalement pour la vérifier avant de la copier.
Vous préférez une copie locale ? Téléchargez les fichiers actuellement disponibles dans SkillsMP.
Télécharger Zip Téléchargement... Métiers associés SOC
Basé sur la classification professionnelle SOC
name mact-document-understanding title MACT - Multi-Agent for Visual Document Understanding with Test-Time Scaling version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2508.03404 keywords ["document-understanding","multi-agent","test-time-scaling","vision-language"] description Decomposes document processing into specialized agents (planning, execution, judgment, answer) with agent-wise adaptive test-time scaling. Achieves 9.9-11.5% performance gain with smaller models while maintaining reasoning.
MACT: Multi-Agent for Visual Document Understanding with Test-Time Scaling
Core Concept
Vision-Language Models struggle with document understanding because documents require procedural reasoning, cognitive complexity, and factual accuracy. Rather than scaling model size, MACT decomposes document processing into specialized agents that collaborate through adaptive test-time scaling. A planning agent breaks tasks into steps, execution agents process content, a judgment agent verifies results, and an answer agent synthesizes responses. Crucially, each agent receives adaptive compute based on task difficulty.
Architecture Overview
Planning Agent : Decomposes document task into reasoning stepsExecution Agents : Process specific document components (text extraction, layout, tables)Judgment Agent : Verifies intermediate results and flags issuesAnswer Agent : Synthesizes final response with self-correctionAdaptive Test-Time Scaling : Allocate more computation to difficult subtasks
Implementation Steps
Step 1: Build Planning Agent
Create agent that decomposes document tasks strategically.
from typing import Dict , List , Tuple
from dataclasses import dataclass
@dataclass
class DocumentTask :
"""Represents a document understanding task."""
document_image: str
query: str
task_type: str
complexity: float = None
class PlanningAgent :
"""
Decomposes complex document task into procedural steps.
"""
def __init__ (self, planning_model ):
self
def
decompose_task
self, task: DocumentTask
List
Dict
"""
Break document task into procedural steps.
Args:
task: Document understanding task
Returns:
List of procedural steps with context
"""
f"""
Document Query: {task.query}
Task Type: {task.task_type}
Break this document task into 3-8 concrete procedural steps.
For each step:
1. What specifically needs to be done?
2. What information from the document is needed?
3. What are success criteria?
Return as JSON list of steps.
"""
self
self
for
in
"complexity"
self
return
def
_estimate_step_complexity
self, step: Dict
float
"""
Estimate computational complexity of step.
Args:
step: Step description
Returns:
Complexity 0-1
"""
"complex"
0.8
"detailed"
0.6
"comprehensive"
0.7
"compare"
0.7
"analyze"
0.6
"extract"
0.3
"identify"
0.4
"description"
""
0.5
for
in
if
in
max
return
def
_parse_steps
self, response: str
List
Dict
"""Parse steps from model response."""
import
import
match
r'\[.*\]'
if
match
return
match
return
Step 2: Build Specialized Execution Agents Create agents optimized for specific document processing tasks.
class ExecutionAgent :
"""
Base execution agent for document processing subtasks.
"""
def __init__ (self, model, agent_type: str ):
self .model = model
self .agent_type = agent_type
self .confidence_scores = []
def execute (self, task_step: Dict , document, context: Dict ) -> Dict :
"""
Execute single procedural step on document.
Args:
task_step: Step to execute
document: Document image/content
context: Prior results/context
Returns:
Execution result with output and confidence
"""
prompt = self ._build_prompt(task_step, context)
output = self .model.generate(prompt)
confidence = self ._estimate_confidence(output, task_step)
self .confidence_scores.append(confidence)
return {
"agent_type" : self .agent_type,
"step" : task_step,
"output" : output,
"confidence" : confidence,
"requires_verification" : confidence < 0.7
}
def _build_prompt (self, step: Dict , context: Dict ) -> str :
"""Build prompt for execution step."""
prompt = f"""
Task: {step.get('description' , '' )}
Prior context: {context}
Complete this task on the document.
Be specific and factual.
"""
return prompt
def _estimate_confidence (self, output: str , step: Dict ) -> float :
"""
Estimate confidence in output.
Returns:
Confidence 0-1
"""
confidence = 0.5
if len (output.split()) > 5 :
confidence += 0.2
uncertainty_words = ["maybe" , "possibly" , "unclear" , "uncertain" ]
if not any (word in output.lower() for word in uncertainty_words):
confidence += 0.3
return min (confidence, 1.0 )
class TextExtractionAgent (ExecutionAgent ):
"""Specialized for text extraction from documents."""
def __init__ (self, model ):
super ().__init__(model, "text_extraction" )
def _build_prompt (self, step: Dict , context: Dict ) -> str :
"""Build text extraction prompt."""
return f"""
Extract the following text from the document:
{step.get('extraction_target' , '' )}
Return only the extracted text, no explanations.
"""
class TableAnalysisAgent (ExecutionAgent ):
"""Specialized for table and structured data analysis."""
def __init__ (self, model ):
super ().__init__(model, "table_analysis" )
def _build_prompt (self, step: Dict , context: Dict ) -> str :
"""Build table analysis prompt."""
return f"""
Analyze the table in the document:
{step.get('table_task' , '' )}
Return analysis as structured JSON.
"""
class LayoutAnalysisAgent (ExecutionAgent ):
"""Specialized for document layout and structure."""
def __init__ (self, model ):
super ().__init__(model, "layout_analysis" )
def _build_prompt (self, step: Dict , context: Dict ) -> str :
"""Build layout analysis prompt."""
return f"""
Analyze the layout and structure of the document:
{step.get('layout_task' , '' )}
Describe the organization and key sections.
"""
Step 3: Build Judgment Agent Create agent that verifies intermediate results.
class JudgmentAgent :
"""
Verifies intermediate results and flags potential issues.
"""
def __init__ (self, model ):
self .model = model
def verify_execution_result (
self,
step: Dict ,
execution_output: str ,
document_context: str ,
confidence_score: float
) -> Dict :
"""
Verify quality of execution result.
Args:
step: Original task step
execution_output: Output from execution agent
document_context: Document content for reference
confidence_score: Execution agent's confidence
Returns:
Verification result with accept/reject and feedback
"""
verification_prompt = f"""
Task: {step.get('description' , '' )}
Execution Output: {execution_output}
Agent Confidence: {confidence_score:.2 f}
Verify this output:
1. Does it complete the task?
2. Is it factually correct based on the document?
3. Is it complete and unambiguous?
Respond with JSON:
{{
"valid": true/false,
"confidence": 0.0-1.0,
"issues": ["issue1", "issue2"],
"feedback": "explanation"
}}
"""
response = self .model.generate(verification_prompt)
result = self ._parse_verification(response)
result["accepted" ] = (
result.get("valid" , False ) and
confidence_score > 0.6 and
result.get("confidence" , 0 ) > 0.7
)
result["requires_refinement" ] = not result["accepted" ]
return result
def _parse_verification (self, response: str ) -> Dict :
"""Parse verification result."""
import json
import re
match = re.search(r'\{.*\}' , response, re.DOTALL)
if match :
return json.loads(match .group())
return {"valid" : False , "confidence" : 0.0 , "issues" : [], "feedback" : "" }
Step 4: Implement Adaptive Test-Time Scaling Create system that allocates compute based on task difficulty.
class AdaptiveTestTimeScaling :
"""
Adaptively allocate computational resources to difficult subtasks.
"""
def __init__ (self, execution_agents: Dict , judgment_agent: JudgmentAgent ):
self .execution_agents = execution_agents
self .judgment = judgment_agent
def execute_with_adaptive_scaling (
self,
task_steps: List [Dict ],
document,
base_compute: int = 1 ,
max_compute: int = 5
) -> List [Dict ]:
"""
Execute steps with adaptive compute allocation.
Args:
task_steps: Procedural steps to execute
document: Document to process
base_compute: Base compute units per step
max_compute: Maximum compute units available
Returns:
List of execution results
"""
results = []
remaining_compute = max_compute
context = {}
for step_idx, step in enumerate (task_steps):
complexity = step.get("complexity" , 0.5 )
step_compute = int (base_compute + (complexity * (max_compute - base_compute)))
step_compute = min (step_compute, remaining_compute)
result = self ._execute_step_with_compute(
step,
document,
context,
compute_budget=step_compute
)
verification = self .judgment.verify_execution_result(
step,
result["output" ],
str (document)[:500 ],
result["confidence" ]
)
if not verification["accepted" ] and remaining_compute > step_compute:
extra_compute = min (step_compute, remaining_compute - step_compute)
result = self ._execute_step_with_compute(
step,
document,
context,
compute_budget=step_compute + extra_compute
)
result["verification" ] = verification
remaining_compute -= step_compute
context[f"step_{step_idx} " ] = result["output" ]
results.append(result)
return results
def _execute_step_with_compute (
self,
step: Dict ,
document,
context: Dict ,
compute_budget: int
) -> Dict :
"""
Execute step with given compute budget.
Higher compute budget = multiple attempts, temperature sampling, etc.
Args:
step: Task step
document: Document
context: Prior context
compute_budget: Compute units available
Returns:
Execution result
"""
agent_type = step.get("agent_type" , "general" )
agent = self .execution_agents.get(agent_type, self .execution_agents.get("general" ))
outputs = []
confidences = []
num_attempts = min (compute_budget, 3 )
for attempt in range (num_attempts):
temperature = 0.3 + (attempt * 0.2 )
agent.model.temperature = temperature
result = agent.execute(step, document, context)
outputs.append(result["output" ])
confidences.append(result["confidence" ])
best_idx = confidences.index(max (confidences))
return {
"output" : outputs[best_idx],
"confidence" : confidences[best_idx],
"num_attempts" : num_attempts,
"all_outputs" : outputs
}
Step 5: Build Answer Synthesis Agent Create agent that synthesizes final answer with self-correction.
class AnswerAgent :
"""
Synthesize final answer from execution results.
"""
def __init__ (self, model ):
self .model = model
def synthesize_answer (
self,
original_query: str ,
execution_results: List [Dict ],
document_context: str
) -> str :
"""
Synthesize final answer from all procedural results.
Args:
original_query: Original document query
execution_results: Results from all execution steps
document_context: Document content
Returns:
Final synthesized answer
"""
execution_summary = self ._summarize_execution_results(execution_results)
synthesis_prompt = f"""
Original Query: {original_query}
Execution Results:
{execution_summary}
Synthesize a comprehensive answer to the original query.
Use only information from the execution results.
If results conflict, note the conflict and provide most reliable answer.
Answer:
"""
answer = self .model.generate(synthesis_prompt)
is_valid = self ._verify_answer_validity(original_query, answer)
if not is_valid:
answer = self ._refine_answer(original_query, answer, execution_summary)
return answer
def _summarize_execution_results (self, results: List [Dict ] ) -> str :
"""Summarize execution results for synthesis."""
summary = ""
for i, result in enumerate (results):
summary += f"Step {i + 1 } : {result['output' ]} \n"
if result.get("verification" ):
if not result["verification" ]["accepted" ]:
summary += f" (Note: Verification flagged issues)\n"
return summary
def _verify_answer_validity (self, query: str , answer: str ) -> bool :
"""Check if answer addresses query."""
return len (answer) > 50
def _refine_answer (self, query: str , initial_answer: str , context: str ) -> str :
"""Refine answer based on verification."""
refinement_prompt = f"""
Query: {query}
Initial Answer: {initial_answer}
Supporting Information: {context}
Improve this answer to better address the query.
"""
refined = self .model.generate(refinement_prompt)
return refined
Practical Guidance
When to Use MACT
Complex document processing : Multi-step reasoning requiredMixed document types : Tables, text, layout analysis neededHigh accuracy requirements : Verification step ensures qualityVariable task complexity : Adaptive scaling handles mixed difficultiesMedium-sized models : Reach large-model performance without scaling
When NOT to Use MACT
Simple document tasks : Single extraction task doesn't need decompositionReal-time constraints : Multi-agent overhead adds latencyStreaming documents : Can't decompose until full document availableVery large documents : Decomposition context may be insufficient
Hyperparameter Recommendations
Base compute units : 1-2 per stepMax compute budget : 3-5 total units per taskComplexity threshold for extra compute : 0.6+Verification confidence threshold : 0.7+Number of execution agents : 3-5 specialized + 1 general
Key Insights The critical insight is recognizing that document understanding isn't monolithic but requires specialized reasoning for different document components. By decomposing tasks, using specialized agents, and adaptively allocating compute to difficult subtasks, MACT achieves strong performance with smaller models. The judgment agent prevents propagation of errors through the pipeline.
Reference Visual Document Understanding via Multi-Agent with Test-Time Scaling (arXiv:2508.03404)
Decomposes document processing into planning, execution, judgment, and answer agents. Introduces agent-wise adaptive test-time scaling that allocates computational resources proportionally to task difficulty. Achieves 9.9-11.5% performance improvement with smaller base models.
