| name | project-fetch-autonomous |
| description | Methodology for evaluating autonomous AI capability progression across phases — measuring when models transition from human-assisted to autonomous execution using robotics tasks as a benchmark. |
| category | ai_collection |
| trigger_words | project fetch, autonomous capability, robotics evaluation, capability progression, phase comparison, robodog, physical world agents |
| date | 2026-07-13T00:00:00.000Z |
| source | Anthropic Research - Project Fetch: Phase two (Jun 18, 2026) |
| url | https://www.anthropic.com/research/project-fetch-phase-two |
Project Fetch: Autonomous Capability Evaluation Framework
Overview
A methodology for evaluating how AI model capabilities progress from human-assisted to autonomous execution, using robotics tasks as a benchmark. The core pattern observed: first, models are helpful to humans. Then, humans are helpful to models. Finally, models are largely able to do things themselves.
Experimental Design
Phase Structure
- Phase 1 (Human teams): Teams of non-expert employees perform tasks with/without AI assistance
- Phase 2 (Autonomous): Latest models attempt same tasks without human assistance, with researcher limited to:
- Plugging in hardware
- Entering initial prompt
- Approving commands
- Approving progression to next task
Task Selection Criteria
- Tasks must be completable by at least one human team in Phase 1
- Tasks should NOT require physical manipulation (e.g., using a controller)
- Tasks should test programmatic control, not low-level robotic actuation
Metrics
- Speed comparison: Time to complete tasks (autonomous vs. human teams)
- Code efficiency: Lines of code generated (autonomous vs. human teams)
- Success rate: Whether tasks are completed correctly on first try
- Error recovery: Ability to work around mistakes and find alternative solutions
Key Findings Pattern
| Capability Stage | Characteristics |
|---|
| Stage 1: Human-helpful | Model assists humans, humans make key decisions |
| Stage 2: Human-helpful-to-model | Humans guide model through difficult steps |
| Stage 3: Autonomous | Model completes tasks independently, 10-37x faster than human teams |
Application to Other Domains
This methodology has been applied to:
- Robotics (Project Fetch): robodog sensor connection, path monitoring, object detection
- Cybersecurity: Previously observed similar progression in red team exercises
The pattern suggests that domains following this progression will eventually see autonomous model capabilities, but NOT necessarily at the level of low-level physical control (e.g., developing specific actuation policies).
Limitations
- Does NOT mean the model has "solved" the domain (e.g., robotics)
- Tasks tested are higher-level programmatic control, not low-level actuation
- Human researcher still provides hardware setup and command approval
- Success on structured tasks doesn't generalize to open-ended physical manipulation