vLLM Tests Speculative Decoding on AMD MI300X/MI355X GPUs

vLLM has benchmarked speculative decoding across AMD Instinct MI300X and MI355X GPUs running the ROCm software stack. The technique attacks a core bottleneck in LLM serving: standard autoregressive decoding commits exactly one token per model pass, forcing generation to advance strictly left-to-right and letting that loop dominate latency on long outputs. Speculative decoding instead splits the work into a cheap draft stage that proposes several future tokens and a verification stage where the original target model checks those candidates in a single pass. Accepted tokens are committed together; the first rejected token is replaced by the target model’s own output, and everything after it is discarded—preserving the target model’s behavior exactly while cutting the number of expensive verification rounds.

The post compares five drafting approaches that differ in how the draft component draws on the target model and whether it generates candidates sequentially, autoregressively, in parallel, or in a hybrid fashion: native MTP built into the model architecture, separate MTP drafter checkpoints, and dedicated target-conditioned networks (EAGLE-3, DFlash, and DSpark). These range from a model-native auxiliary prediction path to standalone speculator models trained against a specific target, with DSpark adding causal correction and confidence-based prefix selection.

The practical takeaway is that speculative decoding is not a guaranteed win. Throughput gains varied widely with the drafting method, proposal length, model family, draft checkpoint, workload, and how often draft tokens were actually accepted. The team frames the results as a tuning and observability problem rather than a drop-in speedup, and documents how to enable each method on AMD hardware—notable as a signal that ROCm is becoming a more viable platform for advanced inference optimizations traditionally showcased on NVIDIA.