advanced-evaluation

name	advanced-evaluation
description	This skill should be used when the user asks to \\"implement LLM-as-judge\\", \\"compare model outputs\\", \\"create evaluation rubrics\\", \\"mitigate evaluation bias\\", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.
type	skill
created	2026-02-27T00:00:00.000Z
domain	software-development
category	testing
risk	safe
source	community
tags	["skill","software-development","testing","evaluation"]

Advanced Evaluation

This skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.

Key insight: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.

When to Use

Activate this skill when:

• Building automated evaluation pipelines for LLM outputs
• Comparing multiple model responses to select the best one
• Establishing consistent quality standards across evaluation teams
• Debugging evaluation systems that show inconsistent results
• Designing A/B tests for prompt or model changes
• Creating rubrics for human or automated evaluation
• Analyzing correlation between automated and human judgments

Core Concepts

The Evaluation Taxonomy

Evaluation approaches fall into two primary categories with distinct reliability profiles:

Direct Scoring: A single LLM rates one response on a defined scale.

• Best for: Objective criteria (factual accuracy, instruction following, toxicity)
• Reliability: Moderate to high for well-defined criteria
• Failure mode: Score calibration drift, inconsistent scale interpretation

Pairwise Comparison: An LLM compares two responses and selects the better one.

• Best for: Subjective preferences (tone, style, persuasiveness)
• Reliability: Higher than direct scoring for preferences
• Failure mode: Position bias, length bias

Research from the MT-Bench paper (Zheng et al., 2023) establishes that pairwise comparison achieves higher agreement with human judges than direct scoring for preference-based evaluation, while direct scoring remains appropriate for objective criteria with clear ground truth.

The Bias Landscape

LLM judges exhibit systematic biases that must be actively mitigated:

Position Bias: First-position responses receive preferential treatment in pairwise comparison. Mitigation: Evaluate twice with swapped positions, use majority vote or consistency check.

Length Bias: Longer responses are rated higher regardless of quality. Mitigation: Explicit prompting to ignore length, length-normalized scoring.

Self-Enhancement Bias: Models rate their own outputs higher. Mitigation: Use different models for generation and evaluation, or acknowledge limitation.