Publication Type

Conference Proceeding Article

Version

acceptedVersion

Publication Date

7-2026

Abstract

Speculative decoding accelerates inference for (M)LLMs, yet a training-decoding discrepancy persists: while existing methods optimize single greedy trajectories, decoding involves verifying and ranking multiple sampled draft paths. We propose Variational Speculative Decoding (VSD), formulating draft training as variational inference over latent proposals (draft paths). VSD maximizes the marginal probability of target-model acceptance, yielding an ELBO that promotes high-quality latent proposals while minimizing divergence from the target distribution. To enhance quality and reduce variance, we incorporate a path-level utility and optimize via an Expectation-Maximization procedure. The E-step draws MCMC samples from an oracle-filtered posterior, while the M-step maximizes weighted likelihood using Adaptive Rejection Weighting (ARW) and Confidence-Aware Regularization (CAR). Theoretical analysis confirms that VSD increases expected acceptance length and speedup. Extensive experiments across LLMs and MLLMs show that VSD achieves up to a 9.58% speedup over EAGLE-3 and 8.80% over ViSpec, significantly improving decoding efficiency.

Discipline

Artificial Intelligence and Robotics

Research Areas

Intelligent Systems and Optimization

Areas of Excellence

Digital transformation

Publication

Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), Seoul, South Korea, July 6-11

First Page

1

Last Page

22

Publisher

IMLS

City or Country

Seoul, Korea

Additional URL

https://icml.cc/virtual/2026/poster/66012

Download

Share

COinS