Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution

Key points

• We introduce Nucleus Speculative Decoding (NSD), a relaxed verification method that incorporates target-model plausibility into speculative decoding.1
• We theoretically characterize the distributional deviation introduced by our method and show that the single-step error is exactly determined by the draft model's excess probability within the target nucleus.1
• We further derive sequence-level fidelity bounds that quantify how local deviations accumulate over autoregressive decoding.1
• Analysis shows that plausibility-aware verification provides an effective approach for relaxed verification and speculative decoding efficiency.1

Sources (1)

• [1]Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 06:18 AM

  Speculative decoding accelerates autoregressive generation by using a lightweight draft model to propose multiple tokens that are verified by a target model in parallel.

  We introduce Nucleus Speculative Decoding (NSD), a relaxed verification method that incorporates target-model plausibility into speculative decoding.

Extractive summary: sentences quoted from the sources.