KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse

Key points

• KV-cache reuse in large language model serving is commonly limited to exact prompt prefixes, preventing reuse when the same retrieved document appears after different system prompts, conversation histories, or dynamic metadata.1
• The system combines cache-aware segment scheduling, prompt construction, capacity management, workload generation, and runtime monitoring.1
• Experiments on long-context benchmarks show that naive segment reuse largely preserves output quality for single-document question answering and summarization, while exposing clear limitations in cross-document reasoning.1
• Under a Zipfian RAG-style workload in which 20% of requests share a fixed prefix, KV-Repurpose achieves a substantially higher token-level cache hit rate than prefix caching at the same 30 GB cache capacity, 63.9% versus 3.3%, resulting in lower TTFT.1

Sources (1)

• [1]KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 06:17 AM

  We present KV-Repurpose, an open-source, vLLM-based system for developing and evaluating segment-wise KV-cache reuse beyond exact prefixes.

  KV-cache reuse in large language model serving is commonly limited to exact prompt prefixes, preventing reuse when the same retrieved document appears after different system prompts, conversation histories, or dynamic metadata.

Extractive summary: sentences quoted from the sources.