Vision Transformers Learn Word Meaning from Images Alone

Key points

• Can a vision model learn what words mean from pixels alone?1
• We derive word representations from embedding differences between word images and matched controls, then test whether semantically related words are neighbors within each model's embedding space, while controlling for spelling similarity.1
• We find that brand names in the same industry group together, as do nouns from the same semantic domain.1
• Contrastive alignment with either encoder further strengthened DINOv3-H's word-level semantic structure, while leaving the other finetuned vision models largely unchanged.1

Sources (1)

• [1]Vision Transformers Learn Word Meaning from Images Alone
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 09:00 AM

  We show that the two largest DINOv3 vision transformers organize words rendered in images by word meaning, despite being pretrained using self-supervision on images only, without paired captions, class labels, or other language supervision.

  Can a vision model learn what words mean from pixels alone?

Extractive summary: sentences quoted from the sources.