KV cache

Timeline

1. Oct 10, 2026 · Opinion / analysis · 1 source
   48Gb VRAM speed AND quality ! (Qwen 3.8 27B Swift 1.5 W8A16)
   Because sometimes you need both speed AND quality, I made my own Qwen 3.8 27B Swift 1.5 quant.
2. Oct 10, 2026 · Opinion / analysis · 1 source
   Strata with Qwen3.8 Flash Next UD-Q4_K_XL
   Most of the benchmarks I've seen are using IQ2 or IQ3 quants, so I wanted to see how Unsloth's UD-Q4KXL performs instead.
3. Oct 8, 2026 · Research paper · 1 source
   Compile the Table: Query-Calibrated Operator Compression for Tabular In-Context Learning
   We propose QCOC (Query-Calibrated Operator Compression), which exploits the exchangeability and repeated use of in-context examples by compiling their full KV cache once into compact memory shared across subsequent queries.
4. Oct 8, 2026 · Research paper · 1 source
   Memory Forcing: Attendable Mid-Horizon History for Streaming Video Generation
   Autoregressive video diffusion enables causal video streaming without a bidirectional pass over the full clip, but existing few-step systems usually retain only the opening and most recent frames in a fixed-size KV cache.
5. Oct 8, 2026 · Research paper · 1 source
   RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation
   To bridge this gap, we introduce RaReCache, a framework that enables a large target model to decode accurately from a cache prefilled by a much smaller source via selective recomputation.
6. Oct 8, 2026 · Research paper · 1 source
   Read What Matters: Query-Adaptive Quantization for KV Caches
   KV-cache entries are stored before their future queries are known, but each decoding query needs precision in different places.
7. Oct 8, 2026 · Research paper · 1 source
   Bridging KV-Cache Quantization and Linear Attention: From Theory to Pretrained Weight Migration
   KV-cache quantization compresses individual KV entries into discrete codes but retains all entries, whereas linear attention recurrently aggregates multiple historical KV contributions into a fixed-size continuous state but can introduce interference.
8. Oct 7, 2026 · Research paper · 1 source
   ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals
   Looped Transformers improve parameter efficiency by repeatedly applying shared Transformer blocks over multiple recurrent loops, increasing computational depth without increasing the parameter count.
9. Oct 7, 2026 · Research paper · 1 source
   Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV Caches
   We introduce Dual-QK, which uses paired non-orthogonal query and key transforms to address this conflict.
10. Oct 7, 2026 · Research paper · 1 source
    CHASE: Channel-Aligned Structure Exploitation for Geometry-Aware Model Engineering
    Geometric and Spectral Alignment (GSA) characterizes trained networks through spectral concentration, physical-channel alignment, support structure, and changes in singular bases.
11. Oct 6, 2026 · Research paper · 1 source
    A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal Attention
    Recent work has explored replacing attention layers' RoPE positional embeddings with alternative decay-based mechanisms, which can then be used to prune KV-cache during inference.
12. Oct 6, 2026 · Research paper · 1 source
    Visual Memory Attacks Can Persist Through The KV Cache
    We then introduce Persistent Visual Memory Injection (P-VMI), which optimizes images to preserve this adversarial behaviour after they are masked from attention.
13. Oct 6, 2026 · Research paper · 1 source
    SPIN: Shadow Predictive Indexer for Sparse Attention
    We propose SPIN (Shadow Predictive Indexer) to reduce this indexer overhead.
14. Oct 6, 2026 · Research paper · 1 source
    Enabling Dynamic Computation in Looped LMs
    Looped LMs are parameter efficient and promise dynamic computation (saving memory and FLOPs on easy tokens).
15. Oct 6, 2026 · Research paper · 1 source
    Hybrid Latent Attention for Looped Language Models
    We propose Hybrid Latent Attention (HLA), which keeps exact keys and values within a sliding window of W recent tokens and stores each older token as a compact latent that the query of each loop reads directly, without reconstructing keys and values.
16. Oct 6, 2026 · Research paper · 1 source
    ReFold: Training-Free Reversible Inter-Turn Context Folding for Long-Horizon Agents
    To overcome these limitations, we introduce ReFold: a training-free rendering layer that preserves the underlying interaction history while compressing only the model's rendered context.
17. Oct 6, 2026 · Research paper · 1 source
    Persistent Memory in Multi-Agent LLM Inference: What It Costs, What It Buys, and When You Can Tell
    Decomposing long-context inference across cooperating agents bounds the active KV cache per call rather than total evidence, which matters when KV-cache memory binds.
18. Oct 6, 2026 · Research paper · 1 source
    TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Models
    In this work, we propose TRACE (Train-Rollout Quantization Alignment via Compact GuidancE), an FP4 quantization framework for RL training of Mixture-of-Experts (MoE) language models that addresses the limitation of existing FP4 RL methods.
19. Oct 5, 2026 · Open-source release · 1 source
    vllm-project/vllm v0.31.0
    Fast restart: the new vllm preload CLI launches the weight-cache daemon that keeps post-quantized weights resident in GPU memory across engine restarts (#56680), now with data parallelism (#57386), MTP draft models (#57312), a /health endpoint (#58552) and a readiness wait (#58370).
20. Sep 30, 2026 · Research paper · 1 source
    SparseEngine: Sparse-First Inference Engine
    We present SparseEngine, a ground-up, sparse-first inference engine whose shared lifecycle contract lets each method control its KV representation and computation while coordinating state transitions with common serving infrastructure.
21. Sep 29, 2026 · Open-source release · 1 source
    NVIDIA/TensorRT-LLM v1.3.0rc29
    Expose Nemotron-H vision-language LoRA configuration for supported inference paths #19151
22. Sep 28, 2026 · Open-source release · 1 source
    unslothai/unsloth v0.1.900-beta: Laya Decision Models + Library
    Run and serve Decision Models like Laya (open-source Jev) locally
23. Sep 22, 2026 · Open-source release · 1 source
    vllm-project/vllm v0.30.0
    This release features 762 commits from 315 contributors (104 new)!
24. Sep 9, 2026 · Open-source release · 1 source
    vllm-project/vllm v0.29.0
    MRV2 also gained CUDA graph memory profiling for KV cache auto-sizing (#53306), batch-sharded sampling that cuts per-step logits memory by 1/TP (#50465), prompt embeds (#42963), extracthiddenstates speculation (#49811), padded FULL cudagraph dispatch for uniform decode under spec decode (#53407), and DP-sync skipping before EAGLE/MTP draft prefill (#53694).
25. Sep 5, 2026 · Open-source release · 1 source
    sgl-project/sglang v0.5.19
    | Model | Type | PRs | Cookbook |
26. Aug 26, 2026 · Open-source release · 1 source
    vllm-project/vllm v0.28.0
    DeepSeek V4: sparse MLA now works end-to-end for plain decode, MTP, and DSpark speculative decoding (#51538), joined by AMD Quark NVFP4 support (#47972), reasoning-effort prompts and mappings (#50580), sparse top-k metadata kernel optimizations (#52084, #51967), narrowed eager CUDA graph regions (#51430, #52401), and ROCm enablement on gfx11 and gfx950 (#47017, #52212).
27. Aug 21, 2026 · Open-source release · 1 source
    ollama/ollama v0.33.0
    Developers can now easily configure Claude Desktop to seamlessly work with Ollama as a third-party gateway provider.
28. Aug 10, 2026 · Open-source release · 1 source
    vllm-project/vllm v0.27.0
    Kimi K3 support with a full stack landing in one release: core model files and kernels (#50089, #50000), Python (#50093) and Rust (#50104) frontends, AttnRes kernels (#50090), DeepGEMM support (#50458), compressed-tensors quantized checkpoints (#50500), DSpark AR fusion (#50242), and an option to shard the shared expert instead of replicating it (#50656).
29. Jul 27, 2026 · Open-source release · 1 source
    vllm-project/vllm v0.26.0
    New Inkling model family with a full support stack: base modeling (#48799), piecewise CUDA graph support (#48822), Hopper FA4 relative attention (#48858), MTP=1 speculative decoding (#48869), LoRA (#48884), and standard ModelOpt NVFP4 quantization (#48990).
30. Jul 11, 2026 · Open-source release · 1 source
    vllm-project/vllm v0.25.0
    Building on quantized-model support from the previous release, MRv2 is now the standard execution path, with new support for EVS (#46535), realtime embeddings (#46762), prefix caching for Mamba hybrid models (#42406), multimodal-prefix bidirectional attention (#46942), and dynamic speculative decoding compatible with full CUDA graphs (#45953).

Often appears with