Surfing the Activation Wave: Analyzing the Information and Geometry of LLM Hidden States for Speculative Decoding

Key points

• Decoder hidden states from different layers of large language models (LLMs) often encode complementary information.1
• We investigate layer-wise Shannon entropy, aggregate entropy under hidden-state concatenation, and pairwise CKA dynamics across depth, revealing distinct geometric and informational compartments within decoder representations.1
• Motivated by these findings, we propose entropy and CKA-based layer selection algorithms that favor hidden-state sets that are both highly informative and minimally redundant.1
• As part of this framework, we introduce token-entropy filtering, which removes hidden states associated with low predictive-entropy tokens, thereby improving the quality of layer-selection statistics.1

Sources (1)

• [1]Surfing the Activation Wave: Analyzing the Information and Geometry of LLM Hidden States for Speculative Decoding
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 03:38 AM

  In this work, we present a principled framework for multi-layer hidden-state selection for speculative decoding based on two complementary criteria: informativeness, measured through matrix-entropy analysis, and diversity, quantified using centered kernel alignment (CKA).

  Decoder hidden states from different layers of large language models (LLMs) often encode complementary information.

Extractive summary: sentences quoted from the sources.