agent-as-a-judge-evaluation-framework

name	agent-as-a-judge-evaluation-framework
title	Agent-as-a-Judge
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2601.05111
keywords	["Evaluation Systems","Agent Design","LLM Evaluation","Multi-Agent Systems"]
description	Transition from simple LLM-based evaluation to agentic judges that employ planning, tool-augmented verification, multi-agent collaboration, and persistent memory. Survey of sophisticated evaluation paradigms for complex, specialized, and multi-step assessment tasks across diverse domains.

When to Use This Skill

• Complex multi-step evaluation requiring decomposition into subtasks
• Scenarios needing verification beyond model inference (code execution, theorem proving)
• Assessments where tool access and external data are crucial
• Domains requiring domain-expert involvement or specialization
• Evaluation where context persistence across decisions improves judgment

When NOT to Use This Skill

• Simple classification tasks (LLM-as-Judge is sufficient)
• Real-time latency-critical evaluation scenarios
• Evaluations where tool access creates safety or security risks
• Scenarios with limited computational budget for multi-agent systems

Problem Summary

Traditional LLM-as-a-Judge evaluation suffers from inherent limitations as assessments become increasingly complex, specialized, and multi-step: models exhibit biases, perform only shallow single-pass reasoning, and cannot verify assessments against real-world observations. For example, code correctness evaluation requires execution verification; mathematical proof assessment benefits from formal verification tools; and nuanced evaluations require consultation with domain experts. These limitations compound when evaluation decisions require persistent context or adaptive strategy adjustment.

Solution: Agent-as-a-Judge Framework

Evolve from static LLM evaluation to autonomous agents employing structured planning, tool integration, multi-agent collaboration, and persistent memory.

class AgentAsAJudge:
    def __init__(self, llm_backbone, tools):
        self.llm = llm_backbone
        self.tools = tools  # Code executor, search, theorem prover, etc.
        self.evaluation_history = {}
        self.domain_experts = []

    def evaluate_complex_submission(self, submission, rubric, context=None):
        """Multi-step agentic evaluation with tool verification"""

        # Step 1: Planning - decompose evaluation into subtasks
        evaluation_plan = self.create_evaluation_plan(submission, rubric)
        

        
        subtask_results = {}
         subtask  evaluation_plan.subtasks:
             subtask.requires_code_execution:
                
                result = .tools.execute_code(submission.code, subtask.test_cases)
                subtask_results[subtask.] = result
             subtask.requires_search:
                
                result = .tools.search(subtask.query)
                subtask_results[subtask.] = result
             subtask.requires_expert_review:
                
                result = .domain_experts[subtask.expert_domain].review(submission)
                subtask_results[subtask.] = result

        
         (.domain_experts) > :
            consensus_rating = .aggregate_expert_opinions(subtask_results)
        :
            consensus_rating = .synthesize_results(subtask_results)

        
        .evaluation_history[submission.] = {
            : evaluation_plan,
            : subtask_results,
            : consensus_rating.trace,
            : consensus_rating.score,
            : consensus_rating.confidence
        }

         consensus_rating

     ():
        
        prompt = 
        plan_text = .llm.generate(prompt)
         parse_evaluation_plan(plan_text)

     ():
        
        debate_prompt = 
        synthesis = .llm.generate(debate_prompt)
         parse_synthesis(synthesis)