LapMerge: Differentiable Exact-Budget Token Merging in Linear Time
We introduce LapMerge, a differentiable merging operator that resolves this trade-off for ordered sequences: a learned mass distribution partitions the sequence into exactly $N$ equal-mass segments under a Laplace-kernel density, and prefix/suffix scans aggregate each segment, so both the forward and the backward pass run in $\mathcal{O}(M+N)$ time.
ProofPaper ↗
Key points
- Modern vision and language models process ever longer token sequences, and reducing them is the main lever on downstream cost.
- Token compression, however, faces a fundamental trade-off: selecting tokens discards information, while existing merging methods preserve it only through pairwise token-to-token matching or $MN$ token-to-output interactions.
- The budget can be changed without changing operator parameters, and the learned mass doubles as a token-level map of where the budget is allocated.
- Across vision and vision-language tasks, LapMerge matches or improves on strong token reducers while being up to two orders of magnitude faster at large sequence lengths; it is particularly effective when token order reflects spatial or temporal locality, and works both as a drop-in reduction module and as a trainable architectural component.
Sources (1)
- [1]LapMerge: Differentiable Exact-Budget Token Merging in Linear TimearXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 05:07 PM
We introduce LapMerge, a differentiable merging operator that resolves this trade-off for ordered sequences: a learned mass distribution partitions the sequence into exactly $N$ equal-mass segments under a Laplace-kernel density, and prefix/suffix scans aggregate each segment, so both the forward and the backward pass run in $\mathcal{O}(M+N)$ time.
Modern vision and language models process ever longer token sequences, and reducing them is the main lever on downstream cost.
Extractive summary: sentences quoted from the sources.
