speceyes-speculative-agentic-acceleration

name	speceyes-speculative-agentic-acceleration
title	SpecEyes: Speculative Acceleration for Agentic Multimodal LLMs
version	0.0.3
engine	skillxiv-v0.0.3-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.23483
keywords	["Speculative Execution","Multimodal LLMs","Agent Acceleration","Latency Optimization","Cognitive Gating"]
description	Accelerate agentic multimodal LLMs via speculative execution without sacrificing accuracy. Use a lightweight tool-free MLLM to predict the main model's decisions and pre-compute tool calls before the main model confirms them. Cognitive gating enables the model to self-assess confidence. Achieves 1.1-3.35x speedup with accuracy improvements up to +6.7%. Use when reducing latency in multi-step agentic workflows, have compute budget for a second model, or need to parallelize tool execution with reasoning.
category	Scaling & Efficiency

Core Principle

Agentic multimodal LLMs solve problems by reasoning, calling tools, and incorporating results in loops: perceive image → reason about task → call tool → incorporate result → reason again. This sequential structure creates latency: the model must finish reasoning before initiating tool calls, then must wait for tool results before the next reasoning step.

SpecEyes exploits the heterogeneity of agent workflows: different steps have different complexity requirements. A tool-free MLLM (lightweight, fast) can predict what tool the main model will call before the main model finishes reasoning. While the main model is still thinking, the lightweight model speculatively executes the predicted tool call. If the main model agrees with the prediction, the result is already available, saving latency. If it disagrees, speculation is discarded.

This is speculative execution (borrowed from CPU architecture): bet on what the main processor will do, do it early in parallel, commit if correct, discard if wrong.

Efficiency Architecture

Baseline (Sequential Agent):

1. Main MLLM perceives image, reasons (time: T_reason)
2. Decides on tool (delay until step 3)
3. Calls tool (time: T_tool)
4. Waits for result
5. Reasons about result (time: T_reason)
6. Repeat

Total time: sequential sum of all steps.

SpecEyes (Speculative Agent):

1. Main MLLM perceives image, reasons (T_reason) in parallel with:
   • Lightweight MLLM predicts tool (T_lightweight_predict)
   • Speculatively calls predicted tool (T_tool in parallel)
2. Main MLLM finishes reasoning, confirms tool choice
3. Tool result is either already available (if prediction was correct) or discarded (if wrong)
4. Continue

Speedup Mechanism: Tool execution now overlaps with main reasoning. If predictions are accurate, tool latency is masked. Speedup = 1 / (1 - overlap_fraction).

Empirical Speedup Curves

SpecEyes achieves empirical speedups on three benchmarks:

V*Bench (Visual Agent Benchmark):

• 1.1-2.5x speedup
• Accuracy maintained or improved (+1-2%)
• Agent workflows with clear tool patterns (search, crop, etc.)

HR-Bench (High-Resolution Vision):

• 2.0-3.35x speedup
• Accuracy +2-6.7% improvement
• Longer workflows (more opportunity for speculation to amortize overhead)

POPE (Object Hallucination Evaluation):