llm-as-judge-evaluation

name	llm-as-judge-evaluation
description	Evaluate LLM outputs using frontier models as judges. Use for pairwise model comparison, quality scoring with custom rubrics, and automated evaluation pipelines. Covers position bias mitigation, statistical significance, and generating preference data for DPO/RLHF.
category	llm-tools
version	1.0.0
author	Synthetic Sciences
license	MIT
tags	["Evaluation","LLM-as-Judge","Pairwise Comparison","Quality Assessment","Rubric Design","Model Comparison","Automated Evaluation"]
dependencies	["openai","anthropic","datasets","numpy"]

LLM-as-Judge Evaluation

When to Use This Skill

Use LLM-as-Judge evaluation when you need to:

• Compare a fine-tuned model vs frontier — Does the student beat the teacher on your task?
• Quality gates before deployment — Automated go/no-go on model releases
• Continuous evaluation — Monitor production model quality over time
• Generate preference data — Create (chosen, rejected) pairs for DPO/RLHF training
• Evaluate without ground truth — When exact answers don't exist (creative, open-ended tasks)

When NOT to Use

• Tasks with verifiable answers (math, code execution) — use exact match or unit tests
• Extremely simple classification — use accuracy/F1 directly
• Safety evaluation — use dedicated safety benchmarks, not general judges

Pairwise Comparison

The most reliable LLM-as-judge method. Show a judge two outputs (A and B) and ask which is better.

Basic Implementation

import openai
import json
import random

client = openai.OpenAI()

PAIRWISE_PROMPT = """You are an expert evaluator. Compare two responses to the same prompt.

## Task Context
{task_description}

## User Input
{user_input}

## Response A
{response_a}

## Response B
{response_b}

## Evaluation Criteria
{criteria}

Which response is better? Consider all criteria above.
Return JSON: {{"winner": "A" or "B" or "tie", "reasoning": "brief explanation"}}"""


def pairwise_compare(user_input, response_a, response_b, task_description, criteria,
                     model="gpt-4o", swap_positions=True):
    """Compare two responses with position bias mitigation."""
    results = []

    # First comparison: A=position1, B=position2
    prompt = PAIRWISE_PROMPT.format(
        task_description=task_description,
        user_input=user_input,
        response_a=response_a,
        response_b=response_b,
        criteria=criteria,
    )
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        response_format={: },
        temperature=,
    )
    result1 = json.loads(resp.choices[].message.content)
    results.append(result1[])

     swap_positions:
        
        prompt_swapped = PAIRWISE_PROMPT.(
            task_description=task_description,
            user_input=user_input,
            response_a=response_b,  
            response_b=response_a,  
            criteria=criteria,
        )
        resp2 = client.chat.completions.create(
            model=model,
            messages=[{: , : prompt_swapped}],
            response_format={: },
            temperature=,
        )
        result2 = json.loads(resp2.choices[].message.content)
        
        swapped_winner = {: , : , : }[result2[]]
        results.append(swapped_winner)

    
     ((results)) == :
         results[]