Publication Type

Conference Proceeding Article

Version

acceptedVersion

Publication Date

9-2026

Abstract

Tree-based speculative decoding accelerates autoregressive generation by verifying a branching tree of draft tokens in a single targetmodel forward pass. However, existing methods prioritize maximizing token-level likelihood or the number of accepted tokens while ignoring a critical “efficiency paradox”: the computational overhead of drafting and verifying big trees can grow super-linearly, particularly at scale. This often leads to negative wall-clock speedup when batch sizes increase or hardware saturation limits are reached. To address this, we propose SMART, a System-aware Marginal Analysis framework for Runtime Tree construction. SMART reformulates tree expansion as a hardware-aware optimization problem that directly maximizes end-toend speedup. By applying a principled marginal benefit–cost rule at inference time, SMART expands a node only when its marginal benefit– cost ratio exceeds the tree-level speedup. SMART is training-free and serves as a plug-and-play controller for existing frameworks like MSD and EAGLE. Extensive evaluations across three MLLMs (e.g., LLaVA, Qwen2-VL) and four LLMs (e.g., Llama-3.1, DeepSeek-R1) demonstrate that SMART consistently outperforms state-of-the-art baselines. It delivers an average additional speedup of 20.0% for MLLMs and 15.4% for LLMs across compute-bound batching regimes and diverse GPU architectures without performance loss

Keywords

Speculative decoding, Multimodal large language models

Discipline

Artificial Intelligence and Robotics

Research Areas

Intelligent Systems and Optimization

Areas of Excellence

Digital transformation

Publication

Proceedings of the 19th European Conference on Computer Vision, ECCV 2026, Malmö, Sweden, September 8-12

First Page

1

Last Page

23

Publisher

ECCV

City or Country

Malmö, Sweden

Download

Share

COinS