메뉴
Calibrate a frozen tb-hard task using Harbor Oracle and real-agent rollouts, then distinguish authentic difficulty from ambiguity, flakiness, leakage, or infrastructure failure.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Build or repair a deterministic Harbor task environment with explicit resources, network policy, health checks, and safe solver-visible boundaries.
Implement a legitimate Harbor Oracle solution and validate its behavior against an independent verifier without weakening the task contract.
Red-team a Harbor verifier for answer leakage, reward hacking, shared-container tampering, artifact manipulation, and semantic under-testing.
Create or evolve a canonical Harbor task inside a stable tb-hard case directory. Use after the design brief names observable acceptance criteria and at least three difficulty dimensions.
Audit a calibrated Harbor task for deterministic purchaser export, leakage, evidence completeness, and release readiness.
Research Harbor and Terminal-Bench 2 patterns to design an original tb-hard task. Use before approving a design brief or when checking whether a candidate is too shallow or derivative.
| name | harbor-rollout-calibration |
| description | Calibrate a frozen tb-hard task using Harbor Oracle and real-agent rollouts, then distinguish authentic difficulty from ambiguity, flakiness, leakage, or infrastructure failure. |
| argument-hint | <case_id> |
| user-invocable | true |
| context | fork |
| agent | harbor-rollout-analyst |
| model | opus |
| effort | xhigh |
| allowed-tools | Read Write Edit Glob Grep Bash |
Target case: $ARGUMENTS
Follow tooling/calibration-protocol.v1.yaml. Do not calibrate until the Oracle has full reward, red-team review has no unresolved P0/P1 finding, and public task content is frozen.
Run or analyze Harbor trials with fixed task revision, protocol metadata, and preserved logs. Use make rollouts CASE=<case_id> AGENT=<agent> MODEL=<model> TRIALS=<n> where local execution is available.
Write evidence/<case_id>/calibration/report.md. Report pass rate, runtime, timeout rate, trajectory-depth observations, failure taxonomy, evidence for the intended difficulty dimensions, and one decision: accept, strengthen, clarify, repair, or reject. Low pass rate alone is not evidence of quality.
