Ml LLM As Judge

LLM-as-Judge Evaluation

Last Updated: 2025-10-26

When to Use This Skill

Use LLM-as-judge evaluation when:

• Subjective quality metrics: Evaluating helpfulness, coherence, creativity, tone
• Open-ended generation: Assessing essay writing, creative content, dialogue quality
• No ground truth: Tasks where reference answers don't exist or are insufficient
• Human preference alignment: Approximating human judgments at scale
• Rapid iteration: Quick feedback on prompt or model changes without human annotation
• Multi-dimensional evaluation: Assessing multiple quality aspects simultaneously
• Pairwise comparison: A/B testing between model outputs or prompt variations
• Cost-effective scaling: Replacing expensive human evaluation for certain tasks

Anti-pattern: Using LLM-as-judge for tasks with objective ground truth (use exact match, BLEU, etc. instead). Never rely solely on LLM judges without validation against human judgments.

Core Concepts

Evaluation Paradigms

1. Pointwise Evaluation

• Judge evaluates single output independently
• Assigns absolute score (e.g., 1-5 rating)
• Simple but prone to inconsistent scale usage

2. Pairwise Evaluation

• Judge compares two outputs (A vs B)
• Determines which is better or if tied
• More reliable than pointwise (relative comparison easier than absolute)
• Can suffer from position bias

3. Reference-Guided Evaluation

• Judge has access to reference answer
• Evaluates output quality against reference
• Useful for factual accuracy, task completion

LLM Judge Models (2024-2025)

Prometheus 2 (Fine-tuned Evaluators)

• Models: prometheus-7b-v2.0, prometheus-8x7b-v2.0
• Specialty: Fine-tuned specifically for evaluation tasks
• Strengths: Consistent scoring, explicit rubrics, reduced bias
• Variants: BGB (Best-of-n Greedy + Backtracking) for improved accuracy
• Open source: Full control and transparency

G-Eval (GPT-based)

• Framework: Uses GPT-4 for evaluation with chain-of-thought
• Strengths: High correlation with human judgments, flexible criteria
• Weaknesses: More expensive, potential GPT-4 biases

GPT-4 / Claude 3 Opus (General-purpose)

• Use case: Quick prototyping, high-quality evaluations
• Strengths: Strong reasoning, nuanced judgments
• Weaknesses: Expensive, API-dependent, potential biases

Llama-3-70B-Instruct / Mixtral-8x7B

• Use case: Cost-effective, self-hosted evaluation
• Strengths: Good performance for many tasks, lower cost
• Weaknesses: Lower quality than specialized models

Bias Mitigation Strategies

Position Bias: Judge favors first/second position in pairwise comparisons

• Solution: Swap positions and average scores

Verbosity Bias: Judge favors longer responses

• Solution: Include length-agnostic criteria, normalize by length

Self-Enhancement Bias: Judge favors outputs from same model family

• Solution: Use different model for judging than generation

Scale Inconsistency: Judge uses rating scale inconsistently

• Solution: Use pairwise comparisons or calibration sets

Uncertainty Quantification

Self-Consistency: Sample multiple judgments and measure agreement Confidence Scoring: Ask judge to provide confidence level Multi-Judge Ensemble: Use multiple judges and measure consensus

Implementation Patterns

Pattern 1: Prometheus 2 Pointwise Evaluation

When to use: Fine-tuned evaluator with explicit rubrics

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

class PrometheusEvaluator:
    """Prometheus 2 evaluator for pointwise assessment."""

    def __init__(self, model_name="prometheus-eval/prometheus-7b-v2.0"):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForCausalLM.from_pretrained(
            model_name,
            torch_dtype=torch.bfloat16,
            device_map="auto",
        )

    def evaluate(
        self,
        instruction: str,
        response: str,
        reference_answer: str = None,
        rubric: str = None,
    ) -> dict:
        """
        Evaluate response using Prometheus format.

        Args:
            instruction: The task instruction
            response: The response to evaluate
            reference_answer: Optional reference answer
            rubric: Evaluation rubric (5-point scale by default)

        Returns:
            Dict with score and feedback
        """
        # Default rubric
        if rubric is None:
            rubric = """
1: The response is completely incorrect or irrelevant.
2: The response has major errors or misses key points.
3: The response is acceptable but has some mistakes or lacks detail.
4: The response is good with minor issues.
5: The response is excellent, accurate, and comprehensive.
"""

        # Prometheus prompt format
        if reference_answer:
            prompt = f"""###Task Description:
An instruction (might include an Input inside it), a response to evaluate, and a score rubric representing a evaluation criteria are given.

1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.
2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.
3. The output format should look as follows: \"Feedback: (write a feedback for criteria) [RESULT] (an integer number between 1 and 5)\"

###The instruction to evaluate:
{instruction}

###Response to evaluate:
{response}

###Reference Answer (Score 5):
{reference_answer}

###Score Rubrics:
{rubric}

###Feedback:"""
        else:
            prompt = f"""###Task Description:
An instruction (might include an Input inside it), a response to evaluate, and a score rubric representing a evaluation criteria are given.

1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.
2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.
3. The output format should look as follows: \"Feedback: (write a feedback for criteria) [RESULT] (an integer number between 1 and 5)\"

###The instruction to evaluate:
{instruction}

###Response to evaluate:
{response}

###Score Rubrics:
{rubric}

###Feedback:"""

        # Generate evaluation
        inputs = self.tokenizer(prompt, return_tensors="pt").to(self.model.device)

        with torch.no_grad():
            outputs = self.model.generate(
                **inputs,
                max_new_tokens=512,
                temperature=0.0,
                do_sample=False,
            )

        evaluation = self.tokenizer.decode(outputs[0], skip_special_tokens=True)

        # Extract score and feedback
        feedback = evaluation.split("###Feedback:")[-1].strip()

        # Parse score
        import re
        score_match = re.search(r'\[RESULT\]\s*(\d+)', feedback)
        score = int(score_match.group(1)) if score_match else None

        return {
            "score": score,
            "feedback": feedback.split("[RESULT]")[0].strip() if score else feedback,
            "raw_output": evaluation,
        }

# Usage
evaluator = PrometheusEvaluator()

result = evaluator.evaluate(
    instruction="Explain photosynthesis in simple terms.",
    response="Photosynthesis is the process where plants use sunlight to make food from water and carbon dioxide.",
    reference_answer="Photosynthesis is the process by which plants convert light energy into chemical energy. Plants use sunlight, water, and carbon dioxide to produce glucose (sugar) and oxygen.",
)

print(f"Score: {result['score']}/5")
print(f"Feedback: {result['feedback']}")

Pattern 2: G-Eval Framework

When to use: GPT-4 based evaluation with chain-of-thought

from openai import OpenAI
import json

class GEvalEvaluator:
    """G-Eval framework using GPT-4 with chain-of-thought."""

    def __init__(self, model="gpt-4-turbo-preview"):
        self.client = OpenAI()
        self.model = model

    def evaluate(
        self,
        task_description: str,
        criteria: str,
        response: str,
        context: str = None,
        min_score: int = 1,
        max_score: int = 5,
    ) -> dict:
        """
        Evaluate response using G-Eval with chain-of-thought.

        Args:
            task_description: What the task is
            criteria: What to evaluate (coherence, fluency, etc.)
            response: The response to evaluate
            context: Optional context (e.g., prompt, reference)
            min_score: Minimum score
            max_score: Maximum score

        Returns:
            Dict with score, reasoning, and confidence
        """
        # G-Eval prompt with chain-of-thought
        prompt = f"""You will be given a task description and evaluation criteria. Your job is to evaluate the given response based on the criteria.

Task Description:
{task_description}

Evaluation Criteria:
{criteria}

{f"Context: {context}" if context else ""}

Response to Evaluate:
{response}

Please evaluate step-by-step:
1. First, identify the key aspects of the evaluation criteria
2. Analyze how well the response meets each aspect
3. Provide a final score from {min_score} to {max_score}

Your response MUST be valid JSON in this format:
{{
  "reasoning": "Detailed step-by-step analysis",
  "score": <integer from {min_score} to {max_score}>,
  "confidence": <float from 0.0 to 1.0>
}}
"""

        # Call GPT-4 with JSON mode
        response = self.client.chat.completions.create(
            model=self.model,
            messages=[
                {
                    "role": "system",
                    "content": "You are an expert evaluator. Provide thoughtful, unbiased assessments.",
                },
                {"role": "user", "content": prompt},
            ],
            response_format={"type": "json_object"},
            temperature=0.0,
        )

        result = json.loads(response.choices[0].message.content)

        return result

    def evaluate_with_uncertainty(
        self,
        task_description: str,
        criteria: str,
        response: str,
        num_samples: int = 5,
    ) -> dict:
        """
        Evaluate with uncertainty quantification via self-consistency.

        Args:
            task_description: Task description
            criteria: Evaluation criteria
            response: Response to evaluate
            num_samples: Number of evaluations to sample

        Returns:
            Dict with mean score, std, and all samples
        """
        scores = []
        reasonings = []

        for _ in range(num_samples):
            result = self.evaluate(task_description, criteria, response)
            scores.append(result["score"])
            reasonings.append(result["reasoning"])

        import numpy as np

        return {
            "mean_score": np.mean(scores),
            "std_score": np.std(scores),
            "min_score": np.min(scores),
            "max_score": np.max(scores),
            "all_scores": scores,
            "sample_reasonings": reasonings,
            "confidence": 1.0 - (np.std(scores) / np.mean(scores)) if np.mean(scores) > 0 else 0.0,
        }

# Usage
evaluator = GEvalEvaluator()

# Single evaluation
result = evaluator.evaluate(
    task_description="Summarize a scientific article for general audience",
    criteria="Evaluate based on: 1) Accuracy of scientific content, 2) Clarity for non-experts, 3) Conciseness",
    response="Researchers found that exercise increases brain health by promoting neurogenesis. Regular physical activity can improve memory and reduce dementia risk.",
    context="Original article discusses BDNF protein and hippocampal neurogenesis in exercise studies.",
)

print(f"Score: {result['score']}/5")
print(f"Reasoning: {result['reasoning']}")

# Evaluation with uncertainty
uncertain_result = evaluator.evaluate_with_uncertainty(
    task_description="Summarize a scientific article for general audience",
    criteria="Evaluate based on accuracy, clarity, and conciseness",
    response="Researchers found that exercise increases brain health...",
    num_samples=5,
)

print(f"Mean score: {uncertain_result['mean_score']:.2f} ± {uncertain_result['std_score']:.2f}")
print(f"Confidence: {uncertain_result['confidence']:.2f}")

Pattern 3: Pairwise Comparison with Bias Mitigation

When to use: A/B testing with position bias correction

from openai import OpenAI
import json

class PairwiseJudge:
    """Pairwise comparison with position bias mitigation."""

    def __init__(self, model="gpt-4-turbo-preview"):
        self.client = OpenAI()
        self.model = model

    def compare(
        self,
        instruction: str,
        response_a: str,
        response_b: str,
        criteria: str = "overall quality",
    ) -> dict:
        """
        Compare two responses with bias mitigation.

        Args:
            instruction: The task instruction
            response_a: First response
            response_b: Second response
            criteria: What to compare on

        Returns:
            Dict with winner and reasoning
        """
        # First comparison (A first, B second)
        result_ab = self._single_comparison(
            instruction, response_a, response_b, criteria, "A", "B"
        )

        # Second comparison (B first, A second) to mitigate position bias
        result_ba = self._single_comparison(
            instruction, response_b, response_a, criteria, "B", "A"
        )

        # Aggregate results
        if result_ab["winner"] == result_ba["winner"]:
            # Consistent winner
            winner = result_ab["winner"]
            confidence = "high"
        elif result_ab["winner"] == "tie" or result_ba["winner"] == "tie":
            # One or both say tie
            winner = "tie"
            confidence = "medium"
        else:
            # Disagreement - position bias detected
            winner = "tie"
            confidence = "low"

        return {
            "winner": winner,
            "confidence": confidence,
            "reasoning_ab": result_ab["reasoning"],
            "reasoning_ba": result_ba["reasoning"],
            "position_bias_detected": result_ab["winner"] != result_ba["winner"] and "tie" not in [result_ab["winner"], result_ba["winner"]],
        }

    def _single_comparison(
        self,
        instruction: str,
        response_first: str,
        response_second: str,
        criteria: str,
        first_label: str,
        second_label: str,
    ) -> dict:
        """Single pairwise comparison."""
        prompt = f"""Compare the following two responses to the given instruction based on {criteria}.

Instruction:
{instruction}

Response {first_label}:
{response_first}

Response {second_label}:
{response_second}

Which response is better? Provide your answer as JSON:
{{
  "winner": "{first_label}" or "{second_label}" or "tie",
  "reasoning": "Detailed explanation of why"
}}
"""

        response = self.client.chat.completions.create(
            model=self.model,
            messages=[
                {
                    "role": "system",
                    "content": "You are an objective evaluator. Provide fair, unbiased comparisons.",
                },
                {"role": "user", "content": prompt},
            ],
            response_format={"type": "json_object"},
            temperature=0.0,
        )

        result = json.loads(response.choices[0].message.content)
        return result

# Usage
judge = PairwiseJudge()

result = judge.compare(
    instruction="Write a haiku about spring",
    response_a="Cherry blossoms bloom\nGentle breeze carries petals\nSpring has come again",
    response_b="Flowers are blooming\nIt is springtime outside now\nNature is pretty",
    criteria="poetic quality and imagery",
)

print(f"Winner: {result['winner']}")
print(f"Confidence: {result['confidence']}")
print(f"Position bias detected: {result['position_bias_detected']}")
print(f"Reasoning (A-B): {result['reasoning_ab']}")

Pattern 4: Multi-Dimensional Evaluation

When to use: Assessing multiple quality aspects simultaneously

from typing import List, Dict
from openai import OpenAI
import json

class MultiDimensionalJudge:
    """Evaluate response across multiple dimensions."""

    def __init__(self, model="gpt-4-turbo-preview"):
        self.client = OpenAI()
        self.model = model

    def evaluate(
        self,
        instruction: str,
        response: str,
        dimensions: List[Dict[str, str]],
    ) -> dict:
        """
        Evaluate response across multiple dimensions.

        Args:
            instruction: Task instruction
            response: Response to evaluate
            dimensions: List of {name, description, min_score, max_score} dicts

        Returns:
            Dict with scores per dimension and overall
        """
        # Build dimension descriptions
        dimension_descriptions = []
        for dim in dimensions:
            dimension_descriptions.append(
                f"- {dim['name']}: {dim['description']} (scale: {dim.get('min_score', 1)}-{dim.get('max_score', 5)})"
            )

        prompt = f"""Evaluate the following response across multiple dimensions.

Instruction:
{instruction}

Response:
{response}

Evaluation Dimensions:
{chr(10).join(dimension_descriptions)}

For each dimension:
1. Provide a score within the specified range
2. Explain your reasoning

Return your evaluation as JSON:
{{
  "dimensions": [
    {{
      "name": "dimension_name",
      "score": <score>,
      "reasoning": "explanation"
    }},
    ...
  ],
  "overall_score": <average score>,
  "summary": "overall assessment"
}}
"""

        response_eval = self.client.chat.completions.create(
            model=self.model,
            messages=[
                {
                    "role": "system",
                    "content": "You are an expert evaluator. Assess each dimension independently.",
                },
                {"role": "user", "content": prompt},
            ],
            response_format={"type": "json_object"},
            temperature=0.0,
        )

        result = json.loads(response_eval.choices[0].message.content)

        # Calculate weighted overall score if weights provided
        if any("weight" in dim for dim in dimensions):
            weighted_score = sum(
                dim_result["score"] * next((d["weight"] for d in dimensions if d["name"] == dim_result["name"]), 1.0)
                for dim_result in result["dimensions"]
            ) / sum(dim.get("weight", 1.0) for dim in dimensions)

            result["weighted_overall_score"] = weighted_score

        return result

# Usage
judge = MultiDimensionalJudge()

result = judge.evaluate(
    instruction="Explain quantum entanglement to a high school student",
    response="Quantum entanglement is when two particles are connected in a weird way. If you measure one particle, it instantly affects the other, even if they're far apart. Einstein called it 'spooky action at a distance.' It's like having two magic coins that always land on opposite sides.",
    dimensions=[
        {
            "name": "accuracy",
            "description": "Scientific accuracy of the explanation",
            "min_score": 1,
            "max_score": 5,
            "weight": 2.0,  # Most important
        },
        {
            "name": "clarity",
            "description": "How clear and understandable the explanation is",
            "min_score": 1,
            "max_score": 5,
            "weight": 1.5,
        },
        {
            "name": "engagement",
            "description": "How engaging and interesting the explanation is",
            "min_score": 1,
            "max_score": 5,
            "weight": 1.0,
        },
        {
            "name": "appropriate_level",
            "description": "Whether the explanation is appropriate for high school students",
            "min_score": 1,
            "max_score": 5,
            "weight": 1.5,
        },
    ],
)

print("Dimension scores:")
for dim in result["dimensions"]:
    print(f"  {dim['name']}: {dim['score']}/5 - {dim['reasoning']}")

print(f"\nOverall score: {result['overall_score']:.2f}")
print(f"Weighted score: {result.get('weighted_overall_score', 'N/A'):.2f}")
print(f"Summary: {result['summary']}")

Pattern 5: Expert-in-the-Loop Validation

When to use: Validating LLM judge against human experts

import pandas as pd
from sklearn.metrics import cohen_kappa_score
from typing import List, Dict
import json

class JudgeValidator:
    """Validate LLM judge against human annotations."""

    def __init__(self, llm_judge):
        """
        Args:
            llm_judge: Any judge class with evaluate() method
        """
        self.llm_judge = llm_judge

    def validate_against_humans(
        self,
        test_cases: List[Dict],
        human_annotations: List[int],
    ) -> dict:
        """
        Validate LLM judge against human expert annotations.

        Args:
            test_cases: List of {instruction, response} dicts
            human_annotations: List of human scores (same order as test_cases)

        Returns:
            Validation metrics
        """
        llm_scores = []

        for case in test_cases:
            result = self.llm_judge.evaluate(
                instruction=case["instruction"],
                response=case["response"],
            )
            llm_scores.append(result["score"])

        # Calculate agreement metrics
        from scipy.stats import pearsonr, spearmanr

        # Pearson correlation (linear relationship)
        pearson_corr, pearson_p = pearsonr(llm_scores, human_annotations)

        # Spearman correlation (rank-order relationship)
        spearman_corr, spearman_p = spearmanr(llm_scores, human_annotations)

        # Cohen's Kappa (categorical agreement)
        kappa = cohen_kappa_score(human_annotations, llm_scores)

        # Mean Absolute Error
        mae = sum(abs(h - l) for h, l in zip(human_annotations, llm_scores)) / len(human_annotations)

        # Identify disagreements
        disagreements = [
            {
                "case": test_cases[i],
                "human_score": human_annotations[i],
                "llm_score": llm_scores[i],
                "difference": abs(human_annotations[i] - llm_scores[i]),
            }
            for i in range(len(test_cases))
            if abs(human_annotations[i] - llm_scores[i]) >= 2  # Significant disagreement
        ]

        return {
            "pearson_correlation": pearson_corr,
            "spearman_correlation": spearman_corr,
            "cohens_kappa": kappa,
            "mean_absolute_error": mae,
            "num_significant_disagreements": len(disagreements),
            "disagreement_rate": len(disagreements) / len(test_cases),
            "sample_disagreements": disagreements[:5],  # Top 5 for review
        }

    def calibrate_judge(
        self,
        calibration_set: List[Dict],
        human_scores: List[int],
        target_correlation: float = 0.8,
    ) -> str:
        """
        Generate calibration recommendations based on validation.

        Args:
            calibration_set: Test cases for calibration
            human_scores: Human expert scores
            target_correlation: Desired correlation threshold

        Returns:
            Calibration recommendations
        """
        validation = self.validate_against_humans(calibration_set, human_scores)

        recommendations = []

        if validation["pearson_correlation"] < target_correlation:
            recommendations.append(
                f"Low correlation ({validation['pearson_correlation']:.2f}). Consider:\n"
                "  - Using a more capable judge model (e.g., GPT-4 instead of GPT-3.5)\n"
                "  - Refining evaluation criteria/rubrics\n"
                "  - Adding more examples to the prompt"
            )

        if validation["mean_absolute_error"] > 1.0:
            recommendations.append(
                f"High MAE ({validation['mean_absolute_error']:.2f}). The judge is often off by >1 point.\n"
                "  - Review sample disagreements\n"
                "  - Clarify scoring rubric\n"
                "  - Use reference-guided evaluation if possible"
            )

        if validation["disagreement_rate"] > 0.2:
            recommendations.append(
                f"High disagreement rate ({validation['disagreement_rate']:.1%}).\n"
                "  - Review cases where judge disagrees significantly\n"
                "  - Add calibration examples to prompt\n"
                "  - Consider ensemble of multiple judges"
            )

        if not recommendations:
            recommendations.append(
                "Judge performance is acceptable. Continue monitoring on production data."
            )

        return "\n\n".join(recommendations)

# Usage
from geval_evaluator import GEvalEvaluator  # From Pattern 2

judge = GEvalEvaluator()
validator = JudgeValidator(judge)

# Test cases with human expert scores
test_cases = [
    {
        "instruction": "Explain photosynthesis",
        "response": "Photosynthesis is how plants make food using sunlight...",
    },
    # ... more cases
]

human_scores = [4, 5, 3, 2, 4, 5, 3, 4]  # From expert annotations

# Validate
validation_results = validator.validate_against_humans(test_cases, human_scores)

print(f"Pearson correlation: {validation_results['pearson_correlation']:.3f}")
print(f"Spearman correlation: {validation_results['spearman_correlation']:.3f}")
print(f"Cohen's Kappa: {validation_results['cohens_kappa']:.3f}")
print(f"MAE: {validation_results['mean_absolute_error']:.2f}")

# Get calibration recommendations
recommendations = validator.calibrate_judge(test_cases, human_scores)
print(f"\nRecommendations:\n{recommendations}")

Code Examples

Example 1: Production LLM-as-Judge Pipeline

from typing import List, Dict, Optional
from dataclasses import dataclass
import json
from datetime import datetime

@dataclass
class JudgmentResult:
    """Result from LLM judge."""
    score: float
    reasoning: str
    confidence: float
    judge_model: str
    timestamp: str
    metadata: Dict = None

class ProductionJudgePipeline:
    """Production-ready LLM-as-judge pipeline with logging and monitoring."""

    def __init__(
        self,
        judge_model: str = "gpt-4-turbo-preview",
        enable_uncertainty: bool = True,
        enable_bias_mitigation: bool = True,
        log_path: str = "judge_logs.jsonl",
    ):
        self.judge = GEvalEvaluator(model=judge_model)
        self.pairwise_judge = PairwiseJudge(model=judge_model)
        self.enable_uncertainty = enable_uncertainty
        self.enable_bias_mitigation = enable_bias_mitigation
        self.log_path = log_path

    def judge_single(
        self,
        task_description: str,
        criteria: str,
        response: str,
        context: Optional[str] = None,
    ) -> JudgmentResult:
        """Judge single response with optional uncertainty quantification."""

        if self.enable_uncertainty:
            result = self.judge.evaluate_with_uncertainty(
                task_description=task_description,
                criteria=criteria,
                response=response,
                num_samples=3,
            )

            judgment = JudgmentResult(
                score=result["mean_score"],
                reasoning=result["sample_reasonings"][0],  # First reasoning
                confidence=result["confidence"],
                judge_model=self.judge.model,
                timestamp=datetime.now().isoformat(),
                metadata={
                    "std_score": result["std_score"],
                    "all_scores": result["all_scores"],
                },
            )
        else:
            result = self.judge.evaluate(
                task_description=task_description,
                criteria=criteria,
                response=response,
                context=context,
            )

            judgment = JudgmentResult(
                score=result["score"],
                reasoning=result["reasoning"],
                confidence=result.get("confidence", 1.0),
                judge_model=self.judge.model,
                timestamp=datetime.now().isoformat(),
            )

        # Log judgment
        self._log_judgment(judgment, task_description, response)

        return judgment

    def judge_pairwise(
        self,
        instruction: str,
        response_a: str,
        response_b: str,
        criteria: str = "overall quality",
    ) -> Dict:
        """Judge pairwise comparison with bias mitigation."""

        result = self.pairwise_judge.compare(
            instruction=instruction,
            response_a=response_a,
            response_b=response_b,
            criteria=criteria,
        )

        # Log comparison
        self._log_comparison(result, instruction, response_a, response_b)

        return result

    def batch_evaluate(
        self,
        examples: List[Dict],
        task_description: str,
        criteria: str,
    ) -> List[JudgmentResult]:
        """Evaluate batch of examples."""

        results = []

        for i, example in enumerate(examples):
            judgment = self.judge_single(
                task_description=task_description,
                criteria=criteria,
                response=example["response"],
                context=example.get("context"),
            )

            results.append(judgment)

            # Progress
            if (i + 1) % 10 == 0:
                print(f"Evaluated {i + 1}/{len(examples)} examples")

        # Calculate statistics
        scores = [r.score for r in results]
        avg_score = sum(scores) / len(scores)
        avg_confidence = sum(r.confidence for r in results) / len(results)

        print(f"\nBatch statistics:")
        print(f"  Average score: {avg_score:.2f}")
        print(f"  Average confidence: {avg_confidence:.2f}")
        print(f"  Score range: {min(scores):.2f} - {max(scores):.2f}")

        return results

    def _log_judgment(self, judgment: JudgmentResult, task: str, response: str):
        """Log judgment to file."""
        log_entry = {
            "type": "single_judgment",
            "task": task,
            "response": response,
            "score": judgment.score,
            "confidence": judgment.confidence,
            "judge_model": judgment.judge_model,
            "timestamp": judgment.timestamp,
        }

        with open(self.log_path, "a") as f:
            f.write(json.dumps(log_entry) + "\n")

    def _log_comparison(self, result: Dict, instruction: str, response_a: str, response_b: str):
        """Log pairwise comparison."""
        log_entry = {
            "type": "pairwise_comparison",
            "instruction": instruction,
            "response_a": response_a,
            "response_b": response_b,
            "winner": result["winner"],
            "confidence": result["confidence"],
            "position_bias_detected": result["position_bias_detected"],
            "timestamp": datetime.now().isoformat(),
        }

        with open(self.log_path, "a") as f:
            f.write(json.dumps(log_entry) + "\n")

# Usage
pipeline = ProductionJudgePipeline(
    judge_model="gpt-4-turbo-preview",
    enable_uncertainty=True,
    enable_bias_mitigation=True,
)

# Single judgment
judgment = pipeline.judge_single(
    task_description="Summarize scientific article",
    criteria="Accuracy, clarity, and conciseness",
    response="Researchers found exercise promotes brain health...",
)

print(f"Score: {judgment.score:.2f} (confidence: {judgment.confidence:.2f})")

# Batch evaluation
examples = [
    {"response": "Response 1..."},
    {"response": "Response 2..."},
    # ... more
]

batch_results = pipeline.batch_evaluate(
    examples=examples,
    task_description="Question answering",
    criteria="Correctness and completeness",
)

Anti-Patterns

Anti-Pattern 1: Using LLM Judge for Objective Tasks

Wrong: Using judge for tasks with clear right/wrong answers

# BAD: LLM judge for exact match tasks
judgment = judge.evaluate(
    task="What is 2+2?",
    response="4",
)  # Expensive and unnecessary

Right: Use deterministic metrics

# GOOD: Exact match for objective tasks
def evaluate_math(response, expected):
    return {"score": 1.0 if response.strip() == expected.strip() else 0.0}

Anti-Pattern 2: Ignoring Position Bias

Wrong: Single pairwise comparison

# BAD: Position bias not addressed
winner = judge.compare(response_a, response_b)  # May favor first position

Right: Swap positions and aggregate

# GOOD: Mitigate position bias
result_ab = judge.compare(response_a, response_b)
result_ba = judge.compare(response_b, response_a)

if result_ab["winner"] == result_ba["winner"]:
    winner = result_ab["winner"]  # Consistent
else:
    winner = "tie"  # Disagreement = bias detected

Anti-Pattern 3: Not Validating Against Humans

Wrong: Blind trust in LLM judge

# BAD: No validation
scores = [judge.evaluate(case) for case in test_cases]
# Are these scores reliable? Unknown!

Right: Validate on calibration set

# GOOD: Validate against human experts
validator = JudgeValidator(judge)
validation = validator.validate_against_humans(test_cases, human_scores)

if validation["pearson_correlation"] < 0.7:
    print("WARNING: Low correlation with humans. Review calibration.")

Related Skills

• llm-benchmarks-evaluation.md: Standard benchmarks for objective capability testing
• llm-evaluation-frameworks.md: Arize Phoenix, Braintrust for production evaluation
• rag-evaluation-metrics.md: RAGAS metrics combining LLM-as-judge with retrieval evaluation
• custom-llm-evaluation.md: Domain-specific evaluation metrics and continuous evaluation
• dspy-evaluation.md: DSPy metric functions and prompt optimization

Summary

LLM-as-judge provides scalable, flexible evaluation for subjective quality metrics:

Key Takeaways:

1. Specialized models: Prometheus 2 (fine-tuned), G-Eval (GPT-4 CoT), general-purpose (Claude/GPT-4)
2. Paradigms: Pointwise (absolute), pairwise (comparative), reference-guided (factual)
3. Bias mitigation: Position swapping, verbosity normalization, multi-judge ensemble
4. Uncertainty: Self-consistency sampling, confidence scoring, agreement metrics
5. Validation: Always validate against human experts on calibration set

Best Practices:

• Use pairwise comparison when possible (more reliable than pointwise)
• Always mitigate position bias by swapping positions
• Quantify uncertainty with multi-sample evaluation
• Validate judge against human annotations before production use
• Use specialized models (Prometheus) for consistent, transparent evaluation
• Reserve LLM judges for subjective tasks (use deterministic metrics for objective tasks)

When to combine with other skills:

• Use llm-benchmarks-evaluation.md for objective capability testing (MMLU, HumanEval)
• Use llm-evaluation-frameworks.md to integrate judges with Phoenix/Braintrust pipelines
• Use rag-evaluation-metrics.md for RAG-specific metrics with LLM-as-judge components
• Use custom-llm-evaluation.md for domain-specific rubrics and safety evaluation