Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds

Key points

• Parallel and semi-autoregressive (semi- AR) drafters improve drafting efficiency by proposing an entire block in a single forward pass, but training them raises a new difficulty: the draft distribution for a given position depends on where the decoding round starts, and where rounds start depends on how many tokens earlier rounds accepted.1
• In this work, we develop a theoretical framework for training and evaluating these drafters by representing speculative decoding as a Markov reward process.1
• This formulation yields the Expected Decoding Rounds (EDR) objective, which weights local rejection costs by state occupancies and exactly equals the expected number of decoding rounds.1
• We then derive an exact temporal-difference gradient that supports unbiased stochastic optimization from target-model rollouts.1

Sources (1)

• [1]Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 04:56 PM

  Speculative decoding accelerates large language model inference by using a low-cost draft model to propose tokens that the full-size target model verifies in parallel.

  Parallel and semi-autoregressive (semi- AR) drafters improve drafting efficiency by proposing an entire block in a single forward pass, but training them raises a new difficulty: the draft distribution for a given position depends on where the decoding round starts, and where rounds start depends on how many tokens earlier rounds accepted.

Extractive summary: sentences quoted from the sources.