Enabling Dynamic Computation in Looped LMs

Key points

• However, state-of-the-art open Looped LMs trained with this dynamic computation capability (Ouro models) do not realize it in practice as each loop iteration (depth) requires its own level of KV-cache, necessitating all loop computations.1
• Moreover, Ouro's early-exit prior is enforced on each token equally, which results in static lower-depth like processing of all tokens regardless of difficulty.1
• In this work, we propose a simple "best-available" KV caching strategy that works out-of-the-box, creating a new frontier in the performance vs depth space.1
• Our approach enables up to 30% reduction in FLOPs and KV memory while retaining full-depth performance, showing the true flexibility of Looped LMs. Furthermore, training looped LMs with awareness about this KV caching strategy improves performance and efficiency.1

Sources (1)

• [1]Enabling Dynamic Computation in Looped LMs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 07:09 PM

  Looped LMs are parameter efficient and promise dynamic computation (saving memory and FLOPs on easy tokens).

  However, state-of-the-art open Looped LMs trained with this dynamic computation capability (Ouro models) do not realize it in practice as each loop iteration (depth) requires its own level of KV-cache, necessitating all loop computations.

Extractive summary: sentences quoted from the sources.