DIPrune: Task-Aware Token Pruning with Dual Importance for Efficient Multimodal Language Models

Key points

• Based on our empirical analysis, we have found that this issue arises because salient tokens in shallow layers persistently suppress emerging semantic ones through numerical inertia, leading to premature discarding of signals crucial for deep reasoning.1
• To address the aforementioned issue, from the task-oriented aspects, we first reformulate training-free pruning as a minimization of the distortion in the final task loss and derive a tractable, token-wise upper bound to serve as a surrogate objective.1
• Accordingly, for the implementation, we propose DIPrune, a rank-based framework that employs a dual importance scoring mechanism to jointly optimize intra-layer static feature saliency and inter-layer dynamic semantic evolution.1
• Extensive experiments on LLaVA and Qwen-VL demonstrate that DIPrune consistently achieves state-of-the-art results.1

Sources (1)

• [1]DIPrune: Task-Aware Token Pruning with Dual Importance for Efficient Multimodal Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 01:38 PM

  Recent training-free pruning approaches for Multimodal Large Language Models (MLLMs) effectively cut computational overhead by exploiting visual redundancy or text-vision attention.

  Based on our empirical analysis, we have found that this issue arises because salient tokens in shallow layers persistently suppress emerging semantic ones through numerical inertia, leading to premature discarding of signals crucial for deep reasoning.

Extractive summary: sentences quoted from the sources.