القائمة
Structured SE task evaluation using 15 benchmark definitions from claude-code-harness research
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
Invoke and resume YAML-defined pipelines by name — /pipeline auto-dev runs the full release pipeline
Full Self Driving — autonomous release loop that processes all auto-dev-eligible GitHub issues until none remain, by repeatedly running /pipeline auto-dev then /homework.
On explicit /homework invocation, analyze the current and linked previous sessions, extract mistakes (찐빠), and report them via omcustom-feedback with a confirmation gate. Auto-activation on session cleanup/session-end signals is OPT-IN (default OFF) — requires an explicit project/user directive. Use when explicitly auditing recent work for harness gaps.
hada.io RSS feed monitoring for AI agent/harness articles with automated /scout analysis
Pre-action boundary checking — validates agent tool calls against declared capabilities and task contracts
Auto-detect project context and optimize harness — deactivate unused agents/skills, suggest missing experts, generate project profile
| name | harness-eval |
| description | Structured SE task evaluation using 15 benchmark definitions from claude-code-harness research |
| scope | harness |
| user-invocable | true |
| argument-hint | [--preset all|quick] [--task task-name] |
| effort | high |
| version | 1.0.0 |
Evaluate agent quality using 15 structured software engineering task definitions with quantitative scoring. Based on research from revfactory/claude-code-harness which demonstrated 60% improvement (49.5 → 79.3 points) through structured pre-configuration.
/omcustom:harness-eval # Run all 15 benchmarks
/omcustom:harness-eval --preset quick # Run top 5 high-impact benchmarks
/omcustom:harness-eval --task api-design # Run specific task benchmark
| Dimension | Weight | Description |
|---|---|---|
| Test Coverage | 30% | Unit test count, edge case coverage, assertion quality |
| Architecture Design | 25% | Separation of concerns, dependency management, scalability |
| Error Handling | 25% | Input validation, error propagation, recovery strategies |
| Extensibility | 20% | Plugin points, configuration flexibility, API surface |
| # | Task | Category | Key Evaluation Criteria |
|---|---|---|---|
| 1 | API Design | Architecture | RESTful conventions, versioning, error responses |
| 2 | Data Modeling | Architecture | Schema normalization, relationships, indexing |
| 3 | Authentication Flow | Security | Token management, session handling, OWASP compliance |
| 4 | Test Suite Creation | Quality | Coverage breadth, assertion quality, edge cases |
| 5 | Error Handler | Reliability | Error classification, recovery, user feedback |
| 6 | Logging System | Observability | Structured logging, levels, correlation IDs |
| 7 | Configuration Manager | Operations | Env-based config, validation, secrets handling |
| 8 | CLI Tool | UX | Argument parsing, help text, exit codes |
| 9 | Database Migration | Data | Reversibility, data preservation, zero-downtime |
| 10 | Cache Layer | Performance | Invalidation strategy, TTL, cache-aside pattern |
| 11 | Queue Consumer | Reliability | Idempotency, retry logic, dead letter handling |
| 12 | Middleware Chain | Architecture | Composability, ordering, short-circuiting |
| 13 | File Processor | I/O | Streaming, error recovery, format validation |
| 14 | Webhook Handler | Integration | Signature verification, retry tolerance, idempotency |
| 15 | Rate Limiter | Security | Algorithm choice, distributed state, fairness |
Each task is scored 0-100 across the 4 quality dimensions:
Score = (test_coverage × 0.30) + (architecture × 0.25) + (error_handling × 0.25) + (extensibility × 0.20)
| Score Range | Grade | Interpretation |
|---|---|---|
| 80-100 | A | Production-ready, well-structured |
| 60-79 | B | Functional with minor gaps |
| 40-59 | C | Works but needs improvement |
| 0-39 | D | Significant structural issues |
all (default)Run all 15 tasks. Full evaluation ~45 minutes.
quickRun top 5 high-impact tasks (1, 3, 4, 5, 12). Quick evaluation ~15 minutes.
This skill provides preset rubrics for the evaluator-optimizer pipeline:
/omcustom:harness-eval → loads rubric → evaluator-optimizer executes → scoring → report
The evaluator-optimizer skill's pre_negotiation phase accepts harness-eval rubric dimensions as sprint contract criteria.
Results saved to .claude/outputs/sessions/{YYYY-MM-DD}/harness-eval-{HHmmss}.md with per-task scores and aggregate grade.
Evaluation framework based on research by revfactory/claude-code-harness. Adapted for oh-my-customcode's evaluator-optimizer pipeline with permission.