From What to Which: Decoding Modifier Grounding in Frozen MLLMs
As Multimodal Large Language Models (MLLMs) can describe increasingly complex visual scenes, token-level grounding becomes crucial.
ProofPaper ↗
Key points
- Yet, when an MLLM generates "the yellow banana on the left", established grounding approaches focus on what is in the image ("banana"), overlooking tokens that help describe which instance is meant ("yellow", "left").
- In this work, we ask whether frozen MLLM representations contain decodable grounding information about the referred instance across generated tokens, extending to modifiers such as attributes, spatial expressions, and relational/action terms.
- To address this question, we introduce OTTER, a lightweight supervised probe over frozen MLLM representations that uses Optimal Transport (OT) to align generated tokens with visual regions and produce compact grounding maps.
Sources (1)
- [1]From What to Which: Decoding Modifier Grounding in Frozen MLLMsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 04:53 PM
As Multimodal Large Language Models (MLLMs) can describe increasingly complex visual scenes, token-level grounding becomes crucial.
Yet, when an MLLM generates "the yellow banana on the left", established grounding approaches focus on what is in the image ("banana"), overlooking tokens that help describe which instance is meant ("yellow", "left").
Extractive summary: sentences quoted from the sources.