eval-harness

# Eval Harness Skill Claude Code 会话的正式评估框架，实施评估驱动开发 (eval-driven development, EDD) 原则。 ## 哲学 评估驱动开发将 evals 视为 "AI 开发的单元测试": - 在实施之前定义预期行为 - 在开发过程中持续运行 evals - 随每次变更跟踪回归 - 使用 pass@k 指标进行可靠性测量 ## Eval 类型 ### 能力评估 (Capability Evals) 测试 Claude 是否能做以前不能做的事情： ```markdown [CAPABILITY EVAL: feature-name] Task: Description of what Claude should accomplish Success Criteria: - [ ] Criterion 1 - [ ] Criterion 2 - [ ] Criterion 3 Expected Output: Description of expected result ``` ### 回归评估 (Regression Evals) 确保变更不会破坏现有功能： ```markdown [REGRESSION EVAL: feature-name] Baseline: SHA or checkpoint name Tests: - existing-test-1: PASS/FAIL - existing-test-2: PASS/FAIL - existing-test-3: PASS/FAIL Result: X/Y passed (previously Y/Y) ``` ## 评分器类型 (Grader Types) ### 1. 基于代码的评分器 使用代码进行确定性检查： ```bash # Check if file contains expected pattern grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL" # Check if tests pass npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL" # Check if build succeeds npm run build && echo "PASS" || echo "FAIL" ``` ### 2. 基于模型的评分器 使用 Claude 评估开放式输出： ```markdown [MODEL GRADER PROMPT] Evaluate the following code change: 1. Does it solve the stated problem? 2. Is it well-structured? 3. Are edge cases handled? 4. Is error handling appropriate? Score: 1-5 (1=poor, 5=excellent) Reasoning: [explanation] ``` ### 3. 人类评分器 标记以进行人工审查： ```markdown [HUMAN REVIEW REQUIRED] Change: Description of what changed Reason: Why human review is needed Risk Level: LOW/MEDIUM/HIGH ``` ## 指标 ### pass@k "k 次尝试中至少有一次成功" - pass@1: 第一次尝试成功率 - pass@3: 3 次尝试内成功 - 典型目标: pass@3 > 90% ### pass^k "所有 k 次试验都成功" - 对可靠性的更高要求 - pass^3: 3 次连续成功 - 用于关键路径 ## Eval 工作流 ### 1. 定义 (Before Coding) ```markdown ## EVAL DEFINITION: feature-xyz ### Capability Evals 1. Can create new user account 2. Can validate email format 3. Can hash password securely ### Regression Evals 1. Existing login still works 2. Session management unchanged 3. Logout flow intact ### Success Metrics - pass@3 > 90% for capability evals - pass^3 = 100% for regression evals ``` ### 2. 实施 (Implement) 编写代码以通过定义的 evals。 ### 3. 评估 (Evaluate) ```bash # Run capability evals [Run each capability eval, record PASS/FAIL] # Run regression evals npm test -- --testPathPattern="existing" # Generate report ``` ### 4. 报告 (Report) ```markdown EVAL REPORT: feature-xyz ======================== Capability Evals: create-user: PASS (pass@1) validate-email: PASS (pass@2) hash-password: PASS (pass@1) Overall: 3/3 passed Regression Evals: login-flow: PASS session-mgmt: PASS logout-flow: PASS Overall: 3/3 passed Metrics: pass@1: 67% (2/3) pass@3: 100% (3/3) Status: READY FOR REVIEW ``` ## 集成模式 ### 实施前 (Pre-Implementation) ``` /eval define feature-name ``` 在 `.claude/evals/feature-name.md` 创建 eval 定义文件 ### 实施期间 (During Implementation) ``` /eval check feature-name ``` 运行当前 evals 并报告状态 ### 实施后 (Post-Implementation) ``` /eval report feature-name ``` 生成完整 eval 报告 ## Eval 存储 在项目中存储 evals: ``` .claude/ evals/ feature-xyz.md # Eval definition feature-xyz.log # Eval run history baseline.json # Regression baselines ``` ## 最佳实践 1. **在编码之前定义 evals** - 强制清晰思考成功标准 2. **频繁运行 evals** - 尽早捕获回归 3. **随时间跟踪 pass@k** - 监控可靠性趋势 4. **尽可能使用代码评分器** - 确定性 > 概率性 5. **安全的人工审查** - 永远不要完全自动化安全检查 6. **保持 evals 快速** - 慢的 evals 不会被运行 7. **与代码一起版本化 evals** - Evals 是一等工件 ## 示例: 添加认证 ```markdown ## EVAL: add-authentication ### Phase 1: Define (10 min) Capability Evals: - [ ] User can register with email/password - [ ] User can login with valid credentials - [ ] Invalid credentials rejected with proper error - [ ] Sessions persist across page reloads - [ ] Logout clears session Regression Evals: - [ ] Public routes still accessible - [ ] API responses unchanged - [ ] Database schema compatible ### Phase 2: Implement (varies) [Write code] ### Phase 3: Evaluate Run: /eval check add-authentication ### Phase 4: Report EVAL REPORT: add-authentication ============================== Capability: 5/5 passed (pass@3: 100%) Regression: 3/3 passed (pass^3: 100%) Status: SHIP IT ```