SPIN: Shadow Predictive Indexer for Sparse Attention

Key points

• Indexer-based sparse attention reduces the cost of core attention by passing only a fixed, small number of important tokens to it.1
• SPIN uses lightweight, history-based prediction to identify important KV blocks, avoiding the need to score the full KV cache at every decoding step.1
• SPIN treats KV blocks and speculative decoding as first-class design and implementation considerations.1
• Across extensive evaluations on long-context and agentic benchmarks, SPIN achieves 30-40% sparsity while preserving task quality.1

Sources (1)

• [1]SPIN: Shadow Predictive Indexer for Sparse Attention
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 07:23 PM

  We propose SPIN (Shadow Predictive Indexer) to reduce this indexer overhead.

  Indexer-based sparse attention reduces the cost of core attention by passing only a fixed, small number of important tokens to it.

Extractive summary: sentences quoted from the sources.