Menu
Structured SE task evaluation using 15 benchmark definitions from claude-code-harness research
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Baseado na classificação ocupacional SOC
Pre-action boundary checking — validates agent tool calls against declared capabilities and task contracts
Auto-detect project context and optimize harness — deactivate unused agents/skills, suggest missing experts, generate project profile
Multi-LLM adversarial consensus loop — 3+ LLMs compete to find flaws in designs/specs until unanimous agreement is reached
Monitor Claude Code releases and auto-generate GitHub issues for each new version
Execute OpenAI Codex CLI prompts and return results
YAML-based DAG workflow engine with topological execution and failure strategies
| name | harness-eval |
| description | Structured SE task evaluation using 15 benchmark definitions from claude-code-harness research |
| scope | harness |
| user-invocable | true |
| argument-hint | [--preset all|quick] [--task task-name] |
| effort | high |
| version | 1.0.0 |
Evaluate agent quality using 15 structured software engineering task definitions with quantitative scoring. Based on research from revfactory/claude-code-harness which demonstrated 60% improvement (49.5 → 79.3 points) through structured pre-configuration.
/omcustomcodex:harness-eval # Run all 15 benchmarks
/omcustomcodex:harness-eval --preset quick # Run top 5 high-impact benchmarks
/omcustomcodex:harness-eval --task api-design # Run specific task benchmark
| Dimension | Weight | Description |
|---|---|---|
| Test Coverage | 30% | Unit test count, edge case coverage, assertion quality |
| Architecture Design | 25% | Separation of concerns, dependency management, scalability |
| Error Handling | 25% | Input validation, error propagation, recovery strategies |
| Extensibility | 20% | Plugin points, configuration flexibility, API surface |
| # | Task | Category | Key Evaluation Criteria |
|---|---|---|---|
| 1 | API Design | Architecture | RESTful conventions, versioning, error responses |
| 2 | Data Modeling | Architecture | Schema normalization, relationships, indexing |
| 3 | Authentication Flow | Security | Token management, session handling, OWASP compliance |
| 4 | Test Suite Creation | Quality | Coverage breadth, assertion quality, edge cases |
| 5 | Error Handler | Reliability | Error classification, recovery, user feedback |
| 6 | Logging System | Observability | Structured logging, levels, correlation IDs |
| 7 | Configuration Manager | Operations | Env-based config, validation, secrets handling |
| 8 | CLI Tool | UX | Argument parsing, help text, exit codes |
| 9 | Database Migration | Data | Reversibility, data preservation, zero-downtime |
| 10 | Cache Layer | Performance | Invalidation strategy, TTL, cache-aside pattern |
| 11 | Queue Consumer | Reliability | Idempotency, retry logic, dead letter handling |
| 12 | Middleware Chain | Architecture | Composability, ordering, short-circuiting |
| 13 | File Processor | I/O | Streaming, error recovery, format validation |
| 14 | Webhook Handler | Integration | Signature verification, retry tolerance, idempotency |
| 15 | Rate Limiter | Security | Algorithm choice, distributed state, fairness |
Each task is scored 0-100 across the 4 quality dimensions:
Score = (test_coverage × 0.30) + (architecture × 0.25) + (error_handling × 0.25) + (extensibility × 0.20)
| Score Range | Grade | Interpretation |
|---|---|---|
| 80-100 | A | Production-ready, well-structured |
| 60-79 | B | Functional with minor gaps |
| 40-59 | C | Works but needs improvement |
| 0-39 | D | Significant structural issues |
all (default)Run all 15 tasks. Full evaluation ~45 minutes.
quickRun top 5 high-impact tasks (1, 3, 4, 5, 12). Quick evaluation ~15 minutes.
This skill provides preset rubrics for the evaluator-optimizer pipeline:
/omcustomcodex:harness-eval → loads rubric → evaluator-optimizer executes → scoring → report
The evaluator-optimizer skill's pre_negotiation phase accepts harness-eval rubric dimensions as sprint contract criteria.
For agent or skill benchmarks, enrich the 0-100 quality score with the agent-eval-framework metrics:
| Metric | Source | Use |
|---|---|---|
correctness | benchmark assertions and acceptance criteria | Required before efficiency is considered |
step_ratio | observed steps vs. ideal trajectory | Advisory signal for unnecessary loops |
tool_call_ratio | observed tool calls vs. ideal trajectory | Advisory signal for noisy tool use |
latency_ratio | observed duration vs. ideal trajectory | Advisory signal for runtime regression |
Evaluation order is fixed: correctness first, efficiency second. A benchmark run with failed correctness cannot be rescued by strong efficiency ratios.
Results saved to .codex/outputs/sessions/{YYYY-MM-DD}/harness-eval-{HHmmss}.md with per-task scores and aggregate grade.
Evaluation framework based on research by revfactory/claude-code-harness. Adapted for oh-my-customcodex's evaluator-optimizer pipeline with permission.