The goal of these experiments is to increase throughput (tokens per second) by using a smaller "draft" model or a specialized mechanism to predict multiple tokens, which are then verified in a single pass by the larger "target" model.

1. Performance Summary

Across all tested models, speculative decoding significantly improves throughput compared to the autoregressive baseline. The data tracks speedup (x), tokens per second (tok/s), Mean Accepted Length (MAL), and Acceptance Rate (AR).

Key Observations:

  • The Trade-off: As the number of speculative tokens ($N$) increases, the throughput typically increases up to a point, but the Acceptance Rate (AR) drops. If $N$ is too high, the overhead of rejecting incorrect tokens outweighs the gains.
  • Method Effectiveness:
  • Native MTP (Multi-Token Prediction): Provides stable and consistent gains, typically ranging from 1.2x to 1.8x speedup.
  • DFlash: Capable of very high speedups (often exceeding 2x on models like Kimi-K2.5 and Qwen3.6), though it shows a steeper decline in acceptance rates as $N$ increases.
  • EAGLE-3: Shows strong results for Kimi-K2.5 and MiniMax-M3, often doubling the baseline throughput.
  • Model Specifics:
  • Kimi-K2.5: Benefited significantly from DFlash and EAGLE-3, with DFlash reaching up to 2.68x speedup on MATH500.
  • Qwen 3.5/3.6: Demonstrated high efficiency with Native MTP and DFlash, frequently pushing throughput above 2,500 tok/s.
  • MiniMax-M3: Showed impressive results with EAGLE-3, reaching up to 2.09x speedup on HumanEval.

---

2. Speculative Decoding Methods Explained

The document references five primary drafting methods:

  1. Native MTP: Uses the model's own multi-token prediction heads.
  2. Gemma 4 MTP: A specialized version of MTP tailored for the Gemma 4 architecture.
  3. EAGLE-3: A state-of-the-art speculator that uses hidden states to predict future tokens.
  4. DFlash: A high-throughput drafting method designed for rapid token generation.
  5. DSpark: An adaptive method that sizes the draft-verification budget based on per-request confidence.

---

3. Implementation Guide (vLLM Commands)

To enable these features, the --speculative-config flag is used. Below is a summary of how to configure different methods:

Common Configuration Pattern:

vllm serve <model_path> \
  --tensor-parallel-size <size> \
  --speculative-config '{"model":"<speculator_model>","num_speculative_tokens":<N>,"method":"<method_name>"}'

Examples by Method:

  • Native MTP:

--speculative-config '{"method":"mtp","numspeculativetokens":3}'

  • EAGLE-3:

--speculative-config '{"model":"<eagle3speculator>","numspeculative_tokens":3,"method":"eagle3"}'

  • DFlash:

--speculative-config '{"method":"dflash","model":"<dflashmodel>","numspeculative_tokens":15}'

  • DSpark:

--speculative-config '{"model":"<dsparkmodel>","numspeculative_tokens":7,"method":"dspark"}'

---

4. Hardware & Software Environment

  • Hardware:
  • 8× AMD Instinct™ MI300X (gfx942)
  • 8× AMD Instinct™ MI355X (gfx950) — used specifically for MiniMax-M3.
  • Software Stack:
  • OS: Ubuntu 22.04.5 LTS
  • Runtime: ROCm/HIP 7.2
  • Frameworks: vLLM 0.23.1rc1, PyTorch 2.11.0, Transformers 5.13.1.

5. Final Recommendations for Tuning

The document suggests a specific workflow for optimizing performance:

  1. Start with a supported configuration from the provided lists.
  2. Monitor Acceptance Behavior: If the Acceptance Rate (AR) is very high, increase $N$ (speculative tokens). If it is very low, decrease $N$ to reduce wasted computation.
  3. Match Sweep to Workload: Different datasets (e.g., GSM8K vs. HumanEval) have different predictability; therefore, the optimal $N$ will vary by task.