DLoop: Looped Speculative Decoding

Key points

• With increasingly capable draft models, we find that the target model frequently accepts all tokens produced in a drafting stage.1
• Adaptive draft length methods decide during decoding how many draft tokens precede a verification, but they raise the speedup only for autoregressive draft models.1
• We propose DLoop, a looped form of speculative decoding that adaptively performs multiple drafting stages before verification.1
• Across diverse speculative decoding methods including EAGLE-3, DFlash, Domino, DSpark, and multi-token prediction modules, DLoop improves the wall-clock speedup by 5 to 41 percent while preserving lossless decoding.1

Sources (1)

• [1]DLoop: Looped Speculative Decoding
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 02:55 AM

  Speculative decoding accelerates autoregressive generation in large language models.

  With increasingly capable draft models, we find that the target model frequently accepts all tokens produced in a drafting stage.

Extractive summary: sentences quoted from the sources.