Adaptive Visual Token Reduction for Accelerated Image Understanding

Key points

• Large Vision-Language Models achieve strong VQA performance, but processing high-resolution, information-rich images requires substantial computation, motivating visual token reduction.1
• However, existing methods often prune individual tokens or rely on fixed-size cropping, limiting their ability to preserve spatially structured information such as horizontally or vertically elongated text.1
• ReFIT consists of Relevance-Guided Window Reshaping (RWR) and Instruction-Guided Token Refinement (ITR), where RWR captures instruction-relevant regions by adapting to their spatial characteristics, while ITR further removes unnecessary visual tokens.1
• Experiments on four VQA benchmarks demonstrate that ReFIT improves answer accuracy while reducing computational cost, and qualitative results demonstrate its effectiveness in localizing relevant regions and removing unnecessary visual information.1

Sources (1)

• [1]Adaptive Visual Token Reduction for Accelerated Image Understanding
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 12:37 AM

  To address this limitation, we propose ReFIT, an instruction-guided visual token reduction framework for efficient LVLM inference.

  Large Vision-Language Models achieve strong VQA performance, but processing high-resolution, information-rich images requires substantial computation, motivating visual token reduction.

Extractive summary: sentences quoted from the sources.