Secure Speculative Decoding for Large Language Models
Speculative decoding accelerates inference for a large language model (LLM), referred to as the target model, by first using a smaller model, referred to as the draft model, to generate candidate tokens and then verifying them with the target model for acceptance or rejection.
Key points
- In this work, we bridge this gap by providing the first systematic study of the security implications of speculative decoding.
- Through a large-scale measurement study, we reveal a pronounced security-utility asymmetry: across a wide range of lossy speculative decoding methods, improvements in inference efficiency come at a disproportionately high cost to security, with attack success rates for jailbreak and prompt injection attacks increasing much faster than utility degrades.
- We then propose SecureSD, a new theory-guided speculative decoding method that enhances security while maintaining efficiency and utility.
- Motivated by this insight, SecureSD applies a stricter verification criterion to draft-model tokens at early decoding positions.
Sources (1)
- [1]Secure Speculative Decoding for Large Language ModelsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 04:59 PM
Speculative decoding accelerates inference for a large language model (LLM), referred to as the target model, by first using a smaller model, referred to as the draft model, to generate candidate tokens and then verifying them with the target model for acceptance or rejection.
In this work, we bridge this gap by providing the first systematic study of the security implications of speculative decoding.
Extractive summary: sentences quoted from the sources.