Why VLMs Miss Small Objects, and When Zooming In Is Safe

Key points

• We ask three questions: what limits them, which of these limits better models can remove, and whether the classical way of handling large images, local decomposition, still has a future.1
• We answer them with a theory built on two quantities of the image interface: S, the number of visual tokens across an object's side, and L, the content a call must cover.1
• The limits: a W x H image sent whole within N tokens gives an object of side m at most msqrt(N/(WH)) tokens per side.1
• Assuming only that more tokens per object and less content per call do not hurt on average, splitting an image cannot lower recall if no view zooms out relative to the whole image and views overlap by one object.1

Sources (1)

• [1]Why VLMs Miss Small Objects, and When Zooming In Is Safe
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 02:10 AM

  Vision-language models (VLMs) often miss small objects in large images.

  We ask three questions: what limits them, which of these limits better models can remove, and whether the classical way of handling large images, local decomposition, still has a future.

Extractive summary: sentences quoted from the sources.