zip2zip++: Towards Inference-Time Token Compression with Minimal Quality Loss

Key points

• We show that this quality gap is not a fundamental price of compression but can be substantially closed through targeted improvements to the compression layer and LZW codec, without modifying the underlying pretrained language model.1
• We address these through decoupled input/output dynamic embedding layer, size-aware RoPE positioning, and a modified LZW codec supporting selective merges and multi-view segmentations.1
• Across four models, our recipe improves perplexity on every corpus we evaluated as well as mathematical reasoning accuracy, closing roughly half the quality gap between zip2zip and the uncompressed baseline at a similar compression ratio on general text.1
• This represents a significant step toward lossless adaptive-vocabulary compression for language models.1

Sources (1)

• [1]zip2zip++: Towards Inference-Time Token Compression with Minimal Quality Loss
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 01:52 PM

  Dynamic vocabularies built on-the-fly via Lempel-Ziv-Welch (LZW) compression let a language model represent the same text in fewer tokens, but prior work shows that this efficiency comes at a cost to reasoning and perplexity.

  We show that this quality gap is not a fundamental price of compression but can be substantially closed through targeted improvements to the compression layer and LZW codec, without modifying the underlying pretrained language model.

Extractive summary: sentences quoted from the sources.