Feature Competition and Its Representational Origins in Vision Language Models: Evidence from the Stroop Paradigm
Across models, we observe a persistent bias toward the feature of text in Stroop images: when text and color disagree, models preferentially generate responses centered to the text rather than color.
ProofPaper ↗
Key points
- When you ask a vision-language model (VLM) a question with an image, it answers your question based on the image.
- This problem is precisely studied by the classical Stroop task in psychology: among a set of deliberately designed images whose text and color conflicts (e.g., \textcolor{blue}{RED}), participants are instructed to report the color.
- This representational asymmetry predicts intervention success: steering models toward text-related directions is substantially easier than steering them toward color-related directions.
- Using a classical psychological paradigm, we demonstrate, at both behavioral and representational levels, that distinct features within the same image can exert unequal influence on VLM behavior.
Sources (1)
- [1]Feature Competition and Its Representational Origins in Vision Language Models: Evidence from the Stroop ParadigmarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 07:13 PM
Across models, we observe a persistent bias toward the feature of text in Stroop images: when text and color disagree, models preferentially generate responses centered to the text rather than color.
When you ask a vision-language model (VLM) a question with an image, it answers your question based on the image.
Extractive summary: sentences quoted from the sources.