| name | speceyes-speculative-agentic-acceleration |
| title | SpecEyes: Speculative Acceleration for Agentic Multimodal LLMs |
| version | 0.0.3 |
| engine | skillxiv-v0.0.3-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2603.23483 |
| keywords | ["Speculative Execution","Multimodal LLMs","Agent Acceleration","Latency Optimization","Cognitive Gating"] |
| description | Accelerate agentic multimodal LLMs via speculative execution without sacrificing accuracy. Use a lightweight tool-free MLLM to predict the main model's decisions and pre-compute tool calls before the main model confirms them. Cognitive gating enables the model to self-assess confidence. Achieves 1.1-3.35x speedup with accuracy improvements up to +6.7%. Use when reducing latency in multi-step agentic workflows, have compute budget for a second model, or need to parallelize tool execution with reasoning. |
| category | Scaling & Efficiency |
Core Principle
Agentic multimodal LLMs solve problems by reasoning, calling tools, and incorporating results in loops: perceive image → reason about task → call tool → incorporate result → reason again. This sequential structure creates latency: the model must finish reasoning before initiating tool calls, then must wait for tool results before the next reasoning step.
SpecEyes exploits the heterogeneity of agent workflows: different steps have different complexity requirements. A tool-free MLLM (lightweight, fast) can predict what tool the main model will call before the main model finishes reasoning. While the main model is still thinking, the lightweight model speculatively executes the predicted tool call. If the main model agrees with the prediction, the result is already available, saving latency. If it disagrees, speculation is discarded.
This is speculative execution (borrowed from CPU architecture): bet on what the main processor will do, do it early in parallel, commit if correct, discard if wrong.
Efficiency Architecture
Baseline (Sequential Agent):
- Main MLLM perceives image, reasons (time: T_reason)
- Decides on tool (delay until step 3)
- Calls tool (time: T_tool)
- Waits for result
- Reasons about result (time: T_reason)
- Repeat
Total time: sequential sum of all steps.
SpecEyes (Speculative Agent):
- Main MLLM perceives image, reasons (T_reason) in parallel with:
- Lightweight MLLM predicts tool (T_lightweight_predict)
- Speculatively calls predicted tool (T_tool in parallel)
- Main MLLM finishes reasoning, confirms tool choice
- Tool result is either already available (if prediction was correct) or discarded (if wrong)
- Continue
Speedup Mechanism: Tool execution now overlaps with main reasoning. If predictions are accurate, tool latency is masked. Speedup = 1 / (1 - overlap_fraction).
Empirical Speedup Curves
SpecEyes achieves empirical speedups on three benchmarks:
V*Bench (Visual Agent Benchmark):
- 1.1-2.5x speedup
- Accuracy maintained or improved (+1-2%)
- Agent workflows with clear tool patterns (search, crop, etc.)
HR-Bench (High-Resolution Vision):
- 2.0-3.35x speedup
- Accuracy +2-6.7% improvement
- Longer workflows (more opportunity for speculation to amortize overhead)
POPE (Object Hallucination Evaluation):
