Attention via Black-Box Vector Search

Key points

• Many existing approaches use maximum inner product search (MIPS) to retrieve the heaviest keys, which motivates the following question: given black-box access to a MIPS oracle, how many keys must be retrieved to output an $\varepsilon$-accurate attention estimate?1
• With a single MIPS index, we show that $Θ(\sqrt{n}/\varepsilon)$ retrieved keys are both sufficient and necessary.1
• We then show that if we allow augmentation of keys and queries, we can bypass the above lower bounds: there exists a simple priority-sampling estimator using a single MIPS index and $O(1/\varepsilon^2)$ retrieved keys.1
• When integrated into LLM inference, our algorithms outperform top-$k$ and sampling approaches used in prior work and yield attention approximation that scales favorably to long contexts.1

Sources (1)

• [1]Attention via Black-Box Vector Search
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 02:14 PM

  Sparse attention mechanisms estimate attention over $n$ tokens using a small subset of keys.

  Many existing approaches use maximum inner product search (MIPS) to retrieve the heaviest keys, which motivates the following question: given black-box access to a MIPS oracle, how many keys must be retrieved to output an $\varepsilon$-accurate attention estimate?

Extractive summary: sentences quoted from the sources.