In-Context Learning Can Help Vision-Language Models Overcome Training Priors
Vision-language models (VLMs) learn strong statistical regularities during training, which can make them fail to properly respond to visual evidence when input images violate those regularities.
ProofPaper ↗
Key points
- It is challenging to disambiguate these prior-induced errors from errors caused by the model missing the underlying visual capabilities.
- In this work, we use controlled visual in-context learning to encourage the model to focus on the visual evidence, and demonstrate that this alone is frequently sufficient to correct the errors -- suggesting that the models in fact do possess the relevant visual capabilities.
- Specifically, we show that across five vision-centric tasks, demonstrations yield large gains on prior-conflicting counterfactual examples ($+27.2%$ for Qwen3-VL 235B and $+15.1%$ for Qwen3-VL 32B) while leaving accuracy on prior-aligned examples nearly unchanged.
- Finally, we find that in-context learning is limited when the underlying visual capability is weak or absent.
Sources (1)
- [1]In-Context Learning Can Help Vision-Language Models Overcome Training PriorsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 03:40 AM
Vision-language models (VLMs) learn strong statistical regularities during training, which can make them fail to properly respond to visual evidence when input images violate those regularities.
It is challenging to disambiguate these prior-induced errors from errors caused by the model missing the underlying visual capabilities.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 9, 2026Continually Post-training Small Models via Synthetic Self-distillation
- Oct 8, 2026Unsupervised Multimodal Reward Modeling
- Oct 8, 2026An all-transistor physical-learning transformer can be trained to perform in-context learning
- Oct 8, 2026Compile the Table: Query-Calibrated Operator Compression for Tabular In-Context Learning
- Oct 8, 2026SFT-as-Context Mitigates Forgetting in Supervised Fine-Tuning
- Oct 7, 2026Leaner Transformers Can Easily Learn to Cluster