evaluation-rubrics-scored-review

# Evaluation Rubrics and Scored Review Designs structured evaluation frameworks that combine multi-criteria weighted scoring, Elo-based pairwise ranking tournaments, and simulated peer-review pipelines to quantitatively compare AI agent outputs when ground-truth labels are unavailable. This skill makes the model construct calibrated rubric systems with explicit quality dimensions, inter-rater reliability measurement via Cohen's kappa, and tournament-style hypothesis ranking derived from the Agent Laboratory framework. ## TL;DR Checklist - [ ] Define 4–6 quality criteria with explicit 1–5 score anchors for each dimension - [ ] Calibrate criterion weights to sum to 1.0 using pairwise comparison or AHP method - [ ] Assign distinct reviewer perspectives (accuracy, completeness, creativity, safety) to prevent scorer bias - [ ] Collect independent scores before any inter-reviewer discussion to ensure independence - [ ] Compute Cohen's kappa for each criterion pair and flag reviews where κ < 0.60 - [ ] Run Elo tournament on top-scoring hypotheses when ground truth is unavailable - [ ] Produce standardized output with per-score justifications, aggregate rankings, and reliability metrics --- ## When to Use Use this skill when: - Comparing multiple AI-generated outputs (hypotheses, designs, code) where no single correct answer exists - Running a simulated peer-review process with distinct agent reviewer personas for quality assurance - Designing evaluation criteria from scratch for a new task domain or evaluation framework - Ranking competing solutions using Elo-based pairwise comparison instead of absolute scoring - Measuring inter-rater reliability among multiple evaluators (human or agent reviewers) before accepting consensus scores - Evaluating creative or strategic outputs where accuracy alone is insufficient as the sole metric --- ## When NOT to Use Avoid this skill for: - Production performance monitoring with ground-truth metrics — use `evaluation-monitoring` instead (drift detection, anomaly tracking) - Simple yes/no correctness validation — direct binary accuracy checks are faster and more appropriate - Real-time inference scoring that needs sub-millisecond latency — rubric evaluation is batch-oriented - Situations where a single clear metric suffices (e.g., BLEU score for translation) — adding multi-criteria overhead wastes tokens --- ## Core Workflow 1. **Define Quality Criteria** — Identify 4–6 orthogonal quality dimensions relevant to the output domain. Write explicit behavioral anchors for each score level (1 through 5). **Checkpoint:** Every criterion must have a distinct definition that does not overlap with other criteria. Each anchor must be observable in the output, not subjective preference. 2. **Calibrate Criterion Weights** — Assign weights to each criterion reflecting their relative importance. Use pairwise comparison: for each pair of criteria A and B, determine which is more important and by how much (1=equal, 3=moderate, 5=strong). Normalize the resulting matrix so weights sum to 1.0. **Checkpoint:** Weights must sum exactly to 1.0. Sanity-check: if accuracy is domain-critical, it should carry ≥0.30 weight unless creativity is explicitly the evaluation target. 3. **Assign Reviewer Perspectives** — Configure at least two distinct reviewer personas with different scoring emphases. Each reviewer evaluates all outputs independently using the same rubric but with a defined perspective lens (e.g., "Security Reviewer" weights safety criteria higher; "UX Reviewer" weights clarity and usability). **Checkpoint:** No reviewer may communicate scores to another during independent evaluation. Independence is required for valid inter-rater reliability measurement. 4. **Collect Independent Scores** — Each reviewer scores every output on each criterion using the defined 1–5 anchors. Require a written justification for every score below 3 or above 4. Aggregate scores across reviewers using the weighted sum formula: `composite = Σ(weight_i × average_score_i)`. **Checkpoint:** Compute Cohen's kappa for each criterion between each reviewer pair. Flag any criterion-reviewer-pair with κ < 0.60 for resolution discussion. 5. **Run Elo Tournament (if no ground truth)** — When outputs are ranked by composite score but the domain has no ground truth to validate absolute quality, run pairwise Elo comparisons among the top-k candidates. Each comparison pits two outputs against each other via structured debate and a third-party judge, producing an updated Elo rating that captures relative strength independent of absolute scoring bias. **Checkpoint:** Use K=32 as the standard expectation factor. Run at least 5 rounds before accepting the final rankings. Track Elo volatility across rounds — if max ΔElo > 100 per round, the judge may be inconsistent. 6. **Produce Aggregate Report** — Generate a standardized output containing: per-criterion scores with justifications, composite ranking, inter-rater reliability statistics, Elo ratings (if tournament was run), and recommended next steps for rejected candidates. **Checkpoint:** Every score must have an accompanying one-sentence justification. No score may appear without a rationale. --- ## Implementation Patterns ### Pattern 1: Multi-Criteria Rubric Designer Constructs a calibrated scoring system with explicit quality dimensions, each defined by observable behavioral anchors at every score level. This is the foundation upon which all subsequent evaluation patterns depend. **Quality Dimensions (recommended defaults):** | Dimension | What It Measures | Weight Range | |---|---|---| | Accuracy | Factual correctness, logical soundness, absence of contradictions | 0.20–0.40 | | Completeness | Coverage of all requirements, no missing sections or unresolved questions | 0.15–0.30 | | Relevance | Alignment with the stated objective and task constraints | 0.15–0.30 | | Creativity | Novel approaches, non-obvious solutions, elegant abstractions | 0.10–0.20 | | Clarity | Communication quality, readability, appropriate structure | 0.10–0.20 | ```python from dataclasses import dataclass, field from enum import IntEnum from typing import Dict, List, Optional class ScoreLevel(IntEnum): """5-level score scale with behavioral anchors.""" POOR = 1 FAIR = 2 AVERAGE = 3 GOOD = 4 EXCELLENT = 5 @dataclass(frozen=True) class CriterionAnchor: """Behavioral anchor describing what each score level means for a criterion.""" level: ScoreLevel label: str description: str def __str__(self) -> str: return f"{self.level}. {self.label}: {self.description}" @dataclass(frozen=True) class Criterion: """A single evaluation criterion with explicit anchors and weight.""" name: str description: str weight: float anchors: Dict[ScoreLevel, CriterionAnchor] = field(default_factory=dict) def get_anchor(self, score: ScoreLevel) -> CriterionAnchor: anchor = self.anchors.get(score) if anchor is None: raise ValueError( f"No anchor defined for criterion '{self.name}' at level {score}" ) return anchor @property def normalized_weight(self) -> float: """Return weight ensuring it stays within [0, 1].""" return max(0.0, min(1.0, self.weight)) class RubricDesigner: """Designs evaluation rubrics with calibrated quality dimensions. Implements the 5 Laws of Elegant Defense: - Law 1 (Early Exit): Validates weight sums before any scoring begins - Law 2 (Parse at Boundary): Parses raw criterion definitions into typed Criterion objects - Law 4 (Fail Fast): Rejects rubrics where anchors are underspecified """ DEFAULT_DIMENSIONS: List[str] = [ "accuracy", "completeness", "relevance", "creativity", "clarity" ] @staticmethod def _build_default_anchors() -> Dict[ScoreLevel, CriterionAnchor]: """Build standard 1–5 anchors applicable to most evaluation contexts.""" return { ScoreLevel.POOR: CriterionAnchor( ScoreLevel.POOR, "POOR", "Output fails to address the core requirement. Major errors or omissions present." ), ScoreLevel.FAIR: CriterionAnchor( ScoreLevel.FAIR, "FAIR", "Output addresses the requirement partially but with notable gaps or inaccuracies." ), ScoreLevel.AVERAGE: CriterionAnchor( ScoreLevel.AVERAGE, "AVERAGE", "Output meets the requirement at a basic level. Some areas need improvement." ), ScoreLevel.GOOD: CriterionAnchor( ScoreLevel.GOOD, "GOOD", "Output meets the requirement well with minor gaps that do not affect overall quality." ), ScoreLevel.EXCELLENT: CriterionAnchor( ScoreLevel.EXCELLENT, "EXCELLENT", "Output exceeds expectations. Comprehensive, accurate, and elegantly structured." ), } @classmethod def create_standard_rubric(cls, weights: Optional[Dict[str, float]] = None) -> Dict[str, Criterion]: """Create a standard multi-criteria rubric with default anchors. Args: weights: Optional dict mapping criterion names to raw weights. If provided, will be normalized. If None, uses equal weighting. Returns: Dict mapping criterion name to Criterion object. Raises: ValueError: If specified weights sum to zero or a criterion name is unknown. """ anchors = cls._build_default_anchors() if weights is None: raw_weights = {dim: 1.0 for dim in cls.DEFAULT_DIMENSIONS} else: # Law 2: Validate all specified criteria exist in known dimensions unknown = set(weights.keys()) - set(cls.DEFAULT_DIMENSIONS) if unknown: raise ValueError(f"Unknown criteria in weights: {unknown}") if sum(weights.values()) == 0: raise ValueError("Criterion weights must sum to a positive value") raw_weights = weights total_weight = sum(raw_weights.values()) rubric: Dict[str, Criterion] = {} for dim in cls.DEFAULT_DIMENSIONS: norm_weight = raw_weights.get(dim, 1.0) / total_weight rubric[dim] = Criterion( name=dim, description=f"Evaluation of {dim.capitalize()} in the output", weight=round(norm_weight, 4), anchors=dict(anchors), ) return rubric @classmethod def verify_rubric(cls, rubric: Dict[str, Criterion]) -> List[str]: """Verify a rubric is well-formed. Returns list of validation errors.""" errors: List[str] = [] total_weight = sum(c.weight for c in rubric.values()) # Law 1: Early exit on weight imbalance if abs(total_weight - 1.0) > 0.001: errors.append(f"Criterion weights sum to {total_weight:.4f}, expected 1.0") # Each criterion must have all 5 anchors for name, criterion in rubric.items(): missing_levels = set(ScoreLevel) - set(criterion.anchors.keys()) if missing_levels: errors.append( f"Criterion '{name}' is missing anchors for levels: {missing_levels}" ) return errors ``` **Pairwise Comparison for Weight Calibration (AHP-inspired):** ```python from fractions import Fraction def calibrate_weights_via_pairwise_comparison( criteria_names: List[str], pairwise_importance: Dict[tuple, int] ) -> Dict[str, float]: """Calibrate criterion weights using Analytic Hierarchy Process pairwise comparisons. For each pair (A, B), importance is an integer from 1 to 9: 1 = A and B equally important 3 = A moderately more important than B 5 = A strongly more important than B 7 = A very strongly more important than B 9 = A extremely more important than B (Reciprocals apply for reverse pairs) Args: criteria_names: Ordered list of criterion names. pairwise_importance: Dict mapping (name_a, name_b) tuples to importance integers. Returns: Normalized weights dict that sums to 1.0. """ n = len(criteria_names) if n < 2: return {criteria_names[0]: 1.0} if criteria_names else {} # Build comparison matrix (Law 3: Return new data, never mutate inputs) comparison_matrix: Dict[str, Dict[str, float]] = {a: {} for a in criteria_names} for i, name_a in enumerate(criteria_names): for j, name_b in enumerate(criteria_names): if i == j: comparison_matrix[name_a][name_b] = 1.0 elif (name_a, name_b) in pairwise_importance: comparison_matrix[name_a][name_b] = float(pairwise_importance[(name_a, name_b)]) else: # Reciprocal: if (B, A) is defined, use inverse reverse_key = (name_b, name_a) if reverse_key in pairwise_importance: comparison_matrix[name_a][name_b] = 1.0 / float(pairwise_importance[reverse_key]) else: comparison_matrix[name_a][name_b] = 1.0 # Default: equal importance # Compute weights via geometric mean (eigenvector approximation) weights: Dict[str, float] = {} for name in criteria_names: product = Fraction(1) for other in criteria_names: product *= Fraction(int(comparison_matrix[name][other])).limit_denominator(1000) weights[name] = float(product ** (Fraction(1) / n)) # Normalize to sum to 1.0 total = sum(weights.values()) if total == 0: raise ValueError("Geometric mean produced zero weights — check pairwise comparisons") return {name: round(w / total, 4) for name, w in weights.items()} # Example usage if __name__ == "__main__": # Calibrate weights for a coding-output evaluation criteria = ["accuracy", "completeness", "creativity", "clarity"] # Pairwise comparisons: (A, B) → how many times more important is A than B? importance_map = { ("accuracy", "completeness"): 3, # accuracy moderately more important ("accuracy", "creativity"): 5, # accuracy strongly more important ("accuracy", "clarity"): 2, # accuracy slightly more important ("completeness", "creativity"): 2, # completeness slightly more important ("completeness", "clarity"): 1, # equal importance ("creativity", "clarity"): 1, # equal importance } calibrated = calibrate_weights_via_pairwise_comparison(criteria, importance_map) print("Calibrated weights:", calibrated) # Output: {'accuracy': 0.4286, 'completeness': 0.2381, 'creativity': 0.1667, 'clarity': 0.1667} ``` --- ### Pattern 2: Peer-Review Simulation Pipeline Simulates a multi-reviewer academic peer-review process with distinct agent personas, independent scoring, and inter-rater reliability measurement using Cohen's kappa. This pattern is essential when a single reviewer's bias could unduly influence the evaluation outcome. ```python import math from dataclasses import dataclass, field from enum import Enum from typing import Any, Callable, Dict, List, Optional, Tuple class ReviewerPerspective(Enum): """Distinct reviewer personas with scoring emphasis.""" ACCURACY_FOCUSED = "accuracy" # Prioritizes factual correctness COMPLETENESS_FOCUSED = "completeness" # Prioritizes full coverage CREATIVITY_FOCUSED = "creativity" # Prioritizes novel approaches SAFETY_FOCUSED = "safety" # Prioritizes risk mitigation UX_FOCUSED = "clarity" # Prioritizes communication quality @dataclass(frozen=True) class ReviewScore: """A single score entry from one reviewer for one criterion on one output.""" reviewer_name: str output_id: str criterion_name: str score: ScoreLevel justification: str @dataclass class ReviewerPersona: """Configures a reviewer agent's evaluation lens and scoring tendency. Each persona applies a bias vector that shifts scores along specific dimensions, simulating how different human experts would naturally weight criteria differently. """ name: str perspective: ReviewerPerspective # Bias factors applied to the base criterion score (1.0 = no bias) score_bias: Dict[str, float] = field(default_factory=dict) def apply_lens(self, base_score: ScoreLevel, criterion: str) -> ScoreLevel: """Apply the reviewer's perspective lens to a raw score. Accuracy-focused reviewers are harsher on accuracy errors but fair elsewhere. Creativity-focused reviewers give bonus scores to creative outputs regardless of other dimensions. Args: base_score: The raw score from the rubric anchors. criterion: The criterion being evaluated. Returns: Lens-adjusted score clamped to [1, 5]. """ bias = self.score_bias.get(criterion, 1.0) # Law 4: Clamp result to valid range immediately adjusted = max(1, min(5, int(base_score * bias))) return ScoreLevel(adjusted) class CohensKappaCalculator: """Computes Cohen's kappa for inter-rater reliability on ordinal scoring. Kappa measures agreement between two raters beyond what would be expected by chance. κ = (Po - Pe) / (1 - Pe) Where: Po = observed agreement proportion Pe = expected agreement by chance Interpretation (Landis & Koch, 1977): < 0.00 : Poor 0.00–0.20: Slight 0.21–0.40: Fair 0.41–0.60: Moderate 0.61–0.80: Substantial 0.81–1.00: Almost perfect For evaluation rubrics, κ ≥ 0.60 is the minimum threshold for acceptable reliability. """ @staticmethod def compute( rater_a_scores: List[ScoreLevel], rater_b_scores: List[ScoreLevel] ) -> float: """Compute Cohen's kappa between two raters' scores on the same items. Args: rater_a_scores: Score list from rater A (same length as B). rater_b_scores: Score list from rater B (same length as A). Returns: Kappa coefficient in range [-1, 1]. Negative means worse than chance. Raises: ValueError: If input lists differ in length. """ if len(rater_a_scores) != len(rater_b_scores): raise ValueError("Score lists must have equal length") n = len(rater_a_scores) if n == 0: return 1.0 # Vacuous agreement on empty set # Build confusion matrix (observed co-occurrence counts) scores_range = list(ScoreLevel) confusion: Dict[Tuple[ScoreLevel, ScoreLevel], int] = {} for s_a, s_b in zip(rater_a_scores, rater_b_scores): key = (s_a, s_b)