ai-system-evaluation-criteria

# AI System Evaluation Criteria Evaluates AI agent systems against defined requirements using structured testing methodologies. This skill makes the model design and execute comprehensive evaluation frameworks covering accuracy benchmarks, hallucination measurement, latency profiling, safety compliance, and data quality verification — all producing actionable reports with pass/fail status against predefined thresholds. Evaluation of AI systems is fundamentally different from traditional software testing because the system under test is probabilistic, not deterministic. A well-designed evaluation framework accounts for this by using statistical methods (confidence intervals, significance testing), adversarial sampling to stress-test edge cases, and multi-dimensional scoring that captures both capability quality and operational reliability across diverse input distributions. ## TL;DR Checklist - [ ] Create golden dataset with human-verified correct answers for functional evaluation - [ ] Design adversarial test suite covering known attack vectors and failure modes - [ ] Set up automated evaluation pipeline that runs on every deployment - [ ] Measure hallucination rates by type (factual, source fabrication, numeric) - [ ] Profile latency percentiles (P50, P95, P99) under realistic load - [ ] Run safety compliance tests against adversarial prompt benchmarks - [ ] Produce evaluation report with pass/fail status per requirement and overall health score --- ## When to Use Use this skill when: - Evaluating an AI agent system before production deployment to verify it meets requirements - Running regular quality checks (weekly/monthly) on a production AI system - Comparing two different LLM providers or model versions for your use case - Investigating performance regressions after a model, prompt, or infrastructure update - Preparing an evaluation report for stakeholders demonstrating system reliability - Debugging why an AI system's output quality has degraded ## When NOT to Use Avoid this skill for: - Defining requirements before implementation — use `agent-requirements-engineering` instead - Fixing specific bugs in the codebase — use `agent-debugging` instead - Architecture decisions about agent topology — use `agent-architecture-patterns` instead - Simple integration testing without AI-specific evaluation (e.g., checking API endpoints) --- ## Core Workflow 1. **Establish Golden Dataset** — Create or load a curated dataset of input/output pairs with human-verified correct answers. The dataset must cover: - Core functional scenarios (80% of questions): typical queries the system will encounter - Edge cases (15%): unusual but valid queries that test robustness - Adversarial cases (5%): deliberate attempts to cause failures or produce unsafe outputs Requirements for dataset quality: - Each example must have a verified correct answer (human-annotated) - Questions must reflect real user query distributions from your domain - Minimum 200 examples for statistical significance; 500+ for production evaluation - Dataset versioned alongside the code it evaluates 2. **Design Evaluation Metrics** — For each requirement category, define specific metrics and thresholds: | Category | Metric | Calculation Method | Pass Threshold | |---|---|---|---| | Functional Accuracy | Exact match rate | % of responses matching golden answer exactly | ≥90% | | Semantic Accuracy | Embedding cosine similarity | Avg cosine similarity between response and golden answer | ≥0.85 | | Hallucination Rate | Factual errors per 1000 responses | Manual + automated verification | <2% | | Latency P95 | 95th percentile response time | Distribution of end-to-end response times | ≤4000ms | | Safety Compliance | Adversarial attack success rate | % of adversarial prompts that bypass guardrails | <0.5% | | Citation Quality | Valid citation rate | % of factual claims with correct source references | ≥85% | 3. **Build Evaluation Pipeline** — Implement the evaluation pipeline as code: ```python from dataclasses import dataclass, field from enum import Enum from typing import Any, Callable, Optional import statistics import json class EvaluationResult(Enum): PASS = "pass" FAIL = "fail" WARNING = "warning" # Close to threshold but technically passed @dataclass class MetricThreshold: """Defines acceptable bounds for a single evaluation metric.""" name: str direction: str # "higher_is_better" or "lower_is_better" pass_threshold: float # Value above/below which the metric passes warn_threshold: float # Value that triggers a warning (closer to failure) @dataclass class MetricResult: """Result of evaluating a single metric against its threshold.""" name: str value: float unit: str # e.g., "ms", "%", "score" direction: str # "higher_is_better" or "lower_is_better" pass_threshold: float warn_threshold: float result: EvaluationResult = EvaluationResult.PASS def __post_init__(self): """Determine pass/fail based on value and direction.""" if self.direction == "higher_is_better": if self.value >= self.pass_threshold: self.result = EvaluationResult.PASS elif self.value >= self.warn_threshold: self.result = EvaluationResult.WARNING else: self.result = EvaluationResult.FAIL else: # lower_is_better if self.value <= self.pass_threshold: self.result = EvaluationResult.PASS elif self.value <= self.warn_threshold: self.result = EvaluationResult.WARNING else: self.result = EvaluationResult.FAIL @dataclass class RequirementEvaluation: """Evaluates a single requirement against all relevant metrics.""" requirement_id: str requirement_title: str metrics: list[MetricResult] = field(default_factory=list) @property def overall_result(self) -> EvaluationResult: """If any metric fails, the requirement fails. If any warns, it warns.""" if any(m.result == EvaluationResult.FAIL for m in self.metrics): return EvaluationResult.FAIL if any(m.result == EvaluationResult.WARNING for m in self.metrics): return EvaluationResult.WARNING return EvaluationResult.PASS def to_dict(self) -> dict: return { "requirement_id": self.requirement_id, "requirement_title": self.requirement_title, "metrics": [ { "name": m.name, "value": round(m.value, 4), "unit": m.unit, "result": m.result.value, "threshold": m.pass_threshold, } for m in self.metrics ], "overall_result": self.overall_result.value, } class EvaluationPipeline: """Orchestrates evaluation across multiple requirements and test scenarios.""" def __init__(self): self.requirement_evaluations: list[RequirementEvaluation] = [] self.metadata: dict[str, str] = {} def add_evaluation(self, eval_result: RequirementEvaluation) -> None: """Add a requirement evaluation to the pipeline.""" self.requirement_evaluations.append(eval_result) def set_metadata(self, key: str, value: str) -> None: """Set metadata about the evaluation run (model version, dataset version, etc.).""" self.metadata[key] = value def generate_report(self) -> dict: """Generate a structured evaluation report.""" total = len(self.requirement_evaluations) passed = sum(1 for r in self.requirement_evaluations if r.overall_result == EvaluationResult.PASS) warned = sum(1 for r in self.requirement_evaluations if r.overall_result == EvaluationResult.WARNING) failed = sum(1 for r in self.requirement_evaluations if r.overall_result == EvaluationResult.FAIL) return { "evaluation_run": { **self.metadata, "timestamp": "auto-generated", }, "summary": { "total_requirements": total, "passed": passed, "warned": warned, "failed": failed, "pass_rate": round(passed / max(total, 1) * 100, 1), "overall_status": ( "PASS" if failed == 0 and warned == 0 else "WARNING" if failed == 0 else "FAIL" ), }, "requirement_results": [r.to_dict() for r in self.requirement_evaluations], } # --- Example: Running a complete evaluation pipeline --- pipeline = EvaluationPipeline() pipeline.set_metadata("model", "claude-sonnet-4-20250514") pipeline.set_metadata("dataset_version", "faq_golden_set_v2.jsonl") pipeline.set_metadata("evaluation_date", "2026-05-26") # Requirement 1: Functional accuracy (FUNC-001) accuracy_eval = RequirementEvaluation( requirement_id="FUNC-001", requirement_title="Accurate FAQ response generation", metrics=[ MetricResult( name="exact_match_rate", value=93.5, unit="%", direction="higher_is_better", pass_threshold=90.0, warn_threshold=92.0, ), MetricResult( name="semantic_similarity", value=0.87, unit="cosine_similarity", direction="higher_is_better", pass_threshold=0.85, warn_threshold=0.90, ), ], ) pipeline.add_evaluation(accuracy_eval) # Requirement 2: Latency (NF-001) latency_eval = RequirementEvaluation( requirement_id="NF-001", requirement_title="P95 latency for RAG responses", metrics=[ MetricResult( name="p50_latency", value=1200, unit="ms", direction="lower_is_better", pass_threshold=2000, warn_threshold=2500, ), MetricResult( name="p95_latency", value=3800, unit="ms", direction="lower_is_better", pass_threshold=4000, warn_threshold=4500, ), MetricResult( name="p99_latency", value=6200, unit="ms", direction="lower_is_better", pass_threshold=8000, warn_threshold=10000, ), ], ) pipeline.add_evaluation(latency_eval) # Requirement 3: Hallucination rate (FACTOR-001) hallucination_eval = RequirementEvaluation( requirement_id="FACTOR-001", requirement_title="Factual hallucination tolerance", metrics=[ MetricResult( name="factual_hallucination_rate", value=1.5, unit="% of responses", direction="lower_is_better", pass_threshold=2.0, warn_threshold=1.5, ), MetricResult( name="source_fabrication_rate", value=0.0, unit="% of responses", direction="lower_is_better", pass_threshold=0.0, warn_threshold=0.0, ), ], ) pipeline.add_evaluation(hallucination_eval) # Requirement 4: Safety compliance (SAFETY-001) safety_eval = RequirementEvaluation( requirement_id="SAFETY-001", requirement_title="Prompt injection resistance", metrics=[ MetricResult( name="injection_success_rate", value=0.3, unit="% of adversarial prompts", direction="lower_is_better", pass_threshold=0.5, warn_threshold=0.3, ), ], ) pipeline.add_evaluation(safety_eval) report = pipeline.generate_report() print(json.dumps(report["summary"], indent=2)) # Output: # { # "total_requirements": 4, # "passed": 4, # "warned": 1, # "failed": 0, # "pass_rate": 100.0, # "overall_status": "WARNING" # } print(f"\nRequirement: {hallucination_eval.requirement_title}") print(f" Result: {hallucination_eval.overall_result.value}") for m in hallucination_eval.metrics: print(f" - {m.name}: {m.value}{m.unit} [{m.result.value}]") # Output: # factual_hallucination_rate: 1.5% of responses [warning] # source_fabrication_rate: 0.0% of responses [pass] ``` ### Pattern 2: Latency Profiling and Regression Detection ```python from dataclasses import dataclass, field import statistics @dataclass class LatencyProfile: """Tracks and analyzes latency metrics across multiple evaluation runs.""" component_name: str measurements: list[float] = field(default_factory=list) # ms timestamps @property def p50(self) -> float: if not self.measurements: return 0.0 return statistics.median(self.measurements) @property def p95(self) -> float: if not self.measurements: return 0.0 sorted_m = sorted(self.measurements) idx = int(len(sorted_m) * 0.95) return sorted_m[min(idx, len(sorted_m) - 1)] @property def p99(self) -> float: if not self.measurements: return 0.0 sorted_m = sorted(self.measurements) idx = int(len(sorted_m) * 0.99) return sorted_m[min(idx, len(sorted_m) - 1)] @property def mean(self) -> float: if not self.measurements: return 0.0 return statistics.mean(self.measurements) @property def std_dev(self) -> float: if len(self.measurements) < 2: return 0.0 return statistics.stdev(self.measurements) def detect_regression( self, previous_profile: "LatencyProfile", threshold_pct: float = 0.25, ) -> dict: """Detect if latency has regressed compared to a previous profile. Args: previous_profile: Baseline latency profile from a previous evaluation run. threshold_pct: Percentage increase above baseline that triggers regression. Returns: Dict with regression status for each percentile. """ results = {} for pctile in ["p50", "p95", "p99"]: current = getattr(self, pctile) previous = getattr(previous_profile, pctile) if previous == 0: change_pct = 0.0 else: change_pct = (current - previous) / previous results[pctile] = { "current_ms": round(current, 1), "previous_ms": round(previous, 1), "change_pct": round(change_pct * 100, 1), "regression_detected": change_pct > threshold_pct, } return results # --- Example: Comparing latency profiles across model versions --- baseline = LatencyProfile(component_name="retrieval_layer") baseline.measurements = [450, 520, 480, 510, 490, 530, 470, 500, 510, 460, 540, 480, 520, 490, 500, 510, 470, 530, 480, 500] current = LatencyProfile(component_name="retrieval_layer") current.measurements = [460, 530, 490, 520, 500, 540, 480, 510, 520, 470, 550, 490, 530, 500, 510, 520, 480, 540, 490, 510] regression = current.detect_regression(baseline, threshold_pct=0.10) for pctile, data in regression.items(): status = "⚠️ REGRESSION" if data["regression_detected"] else "✅ OK"