eval-harness

- Define test cases with known-correct outputs - Run agent against each test case - Score: accuracy, completeness, relevance - Compare against baseline performance - Track performance over time ### 2. Skill Quality Testing - Verify skill instructions produce expected outcomes - Test edge cases and boundary conditions - Measure consistency across multiple runs - Check for harmful or incorrect outputs - Validate against ground truth ### 3. Regression Suite - Collection of previously-passing test cases - Run after any agent/skill modification - Flag regressions with before/after comparison - Maintain pass rate threshold (>= 95%) ### 4. Process Verification - End-to-end process execution with known inputs - Verify each phase produces expected outputs - Check task ordering and dependency satisfaction - Measure total execution time ## Quality Scoring ### Accuracy Score (0-100) - Correctness of output vs expected - Partial credit for partially correct outputs - Penalty for hallucinated or fabricated content ### Completeness Score (0-100) - Coverage of required output elements - Missing sections flagged and scored - Bonus for useful additional context ### Consistency Score (0-100) - Run same input 3 times - Compare outputs for semantic similarity - Flag inconsistencies ### Composite Score - (accuracy * 0.4 + completeness * 0.3 + consistency * 0.3) - Threshold: 80 to pass ## When to Use - After creating new agents or skills - After modifying existing agents or skills - Periodic quality audits - Before promoting skills to production ## Agents Used - Used by process-level evaluation orchestrators - No specific agent dependency (evaluates other agents)