| name | capability-horizon-estimator |
| description | Estimate whether an AI model can complete a task and how long it will take, using METR-style time-horizon modeling. Use when scoping agent work, deciding if a task is within reach, planning retries/parallelism, estimating wall-clock time for SWE/MLE/math tasks, or answering "can you do this" / "how long will this take" for autonomous work. |
Capability & Time Horizon Estimator
Estimate what a model can do and how long it takes using the METR time-horizon framework: every task has a human-equivalent duration $t$, every model has a 50% time horizon $h_{50}$, and success probability follows a logistic curve in $\log(h/t)$.
When to use
- "Can you do X autonomously?" — give a calibrated yes/maybe/no instead of vibes
- "How long will this agent run take?" — wall-clock estimate with retry overhead
- Scoping: should this be one task, or decomposed?
- Deciding attempt budget (best-of-N) for a stretch task
The core model
Step 1 — Estimate human-equivalent task time $t$ (minutes).
Anchor against known reference points (full table in references/benchmark-catalog.md):
| Reference task | Human time |
|---|
| SWE-bench Verified issue | 7 min – 2 h |
| HCAST task bands | 15 min / 1 h / 4 h / 8 h |
| RE-Bench ML research task | 8 h |
| Small bug fix, clear repro | 15–60 min |
| Multi-file feature | 2–8 h |
| Cross-repo refactor | 4–16 h |
| Kaggle competition (MLE-bench) | days of human effort |
| FrontierMath T4 problem | days–weeks of expert time |
Estimate for a low-context professional (new hire, contractor), not the resident expert — that is what the horizons are calibrated against.
Step 2 — Get the model's horizon $h_{50}$ (minutes).
Look it up in references/horizon-data.md (METR TH v1.1, May 2026). If the model isn't listed, extrapolate from release date:
$$h_{50}(\text{date}) = h_{50}(\text{ref}) \times 2^{(\text{date} - \text{ref}) / D}, \quad D \approx 130\text{–}190 \text{ days}$$
Use $D = 150$ days as default; state the range. Post-2024 data supports faster ($\sim$90–130 days); all-time average is $\sim$190.
Step 3 — Success probability.
$$p = \sigma!\big(\beta \cdot \ln(h_{50}/t)\big), \quad \beta \approx 0.8 \text{ (range 0.6–0.9)}, \quad \sigma(x) = \frac{1}{1+e^{-x}}$$
Sanity anchors with $\beta = 0.8$: $t = h_{50} \Rightarrow p = 50%$ · $t = h_{50}/5.7 \Rightarrow p = 80%$ · $t = 2h_{50} \Rightarrow p \approx 36%$ · $t = 4h_{50} \Rightarrow p \approx 25%$.
Step 4 — Difficulty adjustments (multiply $t$ before Step 3).
| Condition | Multiplier on $t$ |
|---|
| Well-specified, auto-verifiable (benchmark-like) | ×1 |
| Proprietary/large codebase, high prior context needed | ×2 |
| Messy, underspecified, human-judged success | ×4 (range ×2–8) |
| GUI/computer-use (no API/DOM) | ×50–100 — different capability cluster |
| Parallelizable/additive work (N independent items) | Not one task — split, apply model per piece |
Rationale: METR's suite is clean and self-contained; models do measurably worse on messy tasks and under holistic (vs algorithmic) scoring. GUI horizons measured ~40–100× shorter than SWE/reasoning.
Step 5 — "Pushed itself" uplift.
- Better scaffold/elicitation: ×1.5–2 on $h_{50}$ (scaffold choice moves measured horizons ~2×)
- Best-of-N independent attempts: $P(\geq 1 \text{ success}) = 1 - (1-p)^N$
- Both compose. A $p = 0.25$ task with a good scaffold ($\times 1.75 h$) and 6 attempts: $p \to \sigma(0.8 \ln(1.75 \cdot h/t)) \approx 0.37$ per attempt, $1-(1-0.37)^6 \approx 94%$.
Step 6 — Wall-clock time.
Agents complete tasks they can do several times faster than the human baseline:
$$\text{wall-clock} \approx \frac{t_{\text{human}}}{S} \times \text{expected attempts}, \quad S \approx 3\text{–}8\times \text{ for SWE (default 5)}$$
Expected attempts $= 1/p$ for independent retries (cap at 3–5 in practice; if it fails 3×, the plan is wrong, not the luck). Report a range, not a point.
Output format
TASK: <one line>
HUMAN-EQUIVALENT TIME (t): <estimate> (anchor: <reference task>)
MODEL HORIZON (h50): <value> (source: table / extrapolated from <date>, D=150d)
DIFFICULTY: ×<mult> (<reason>) → effective t = <value>
SINGLE-ATTEMPT SUCCESS: <p> → <verdict>
PUSHED (scaffold ×1.75, N=<n>): <p_eff> per attempt, <P> overall
WALL-CLOCK ESTIMATE: <range> (t/S × attempts, S=5)
VERDICT: CONFIDENT (≥80%) | LIKELY (60–80%) | COIN-FLIP (35–60%) | STRETCH (15–35%) | OUT OF REACH (<15%)
CAVEATS: <extrapolation flags, messiness, jagged-domain warnings>
DECOMPOSITION ADVICE: <if stretch/out-of-reach: where to split>
Calibration rules
- Always state the anchor. Every $t$ estimate names the reference task it was derived from. No anchor, no number.
- Report ranges. Horizons have wide CIs (Opus 4.6 p50: 317–3634 min). Give the interval, pick the geometric mean for the headline.
- Above 16 h human-equivalent, say so. METR's suite saturates at 16 h; estimates beyond it are extrapolation, not measurement.
- Jaggedness is real. A model strong in SWE may be 50–100× weaker on GUI work, and different on math research. Never transfer a horizon across domains without the multiplier.
- Long ≠ hard, hard ≠ long. The model applies to coherent, non-parallelizable tasks. 1000 independent 1-hour problems is a 1-hour task done 1000 times.
- Failure mode asymmetry. Near the horizon, models fail by derailing (repeating failed approaches, losing state), not by slow progress. If $p < 50%$, recommend decomposition over retries.
Reference files
references/horizon-data.md — METR TH v1.1 per-model p50/p80 table (2019 → May 2026), doubling-time stats, methodology caveatsreferences/benchmark-catalog.md — SWE/MLE/math benchmark inventory with task times, human budgets, and July-2026 SOTAscripts/estimate.py — calculator for Steps 3–6 (logistic p, best-of-N, wall-clock)
