Rehearse Everything, Remember Nothing: Attic-KV Rehearses What Will Be Read

Key points

• The prevailing approach scores KV entries by rehearsal: the model rereads the context and keeps the entries it attends to, assuming that the more completely a cache rehearses its context, the better it remembers it.1
• We show that under tight budgets this assumption backfires: rehearse everything, remember nothing.1
• The cause is that a cache keeps what it rehearses: rereading spreads the budget across the whole context, so the answer's own entries survive at little more than chance.1
• We instantiate them as Attic-KV (Attic for short), a training-free rehearsal in which the model quizzes itself with question-answer pairs that quote the context, alongside anchor tokens in a content-adaptive amount.1

Sources (1)

• [1]Rehearse Everything, Remember Nothing: Attic-KV Rehearses What Will Be Read
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 03:22 PM

  Many key-value (KV) caches are compressed before anyone knows what will be asked of them: a document cached for retrieval, a prompt prefix shared across requests, the memory of a long conversation.

  The prevailing approach scores KV entries by rehearsal: the model rereads the context and keeps the entries it attends to, assuming that the more completely a cache rehearses its context, the better it remembers it.

Extractive summary: sentences quoted from the sources.