Learning Features, Not Layouts: Feature-Axis Attention as a Transfer Prior for Medical Imaging

Key points

• Standard vision transformers place all global modeling on self-attention over spatial tokens, an object-layout prior well suited to natural scenes but not to medical images, where the discriminative signal lies in feature correlations and texture that recur across scanners while layout does not.1
• We ask whether the axis of global attention, spatial versus feature, is itself a lever for transferable medical representations.1
• We design FAD-ViT, a backbone that strictly decouples the two axes, making feature-axis attention the primary global operator and the attention axis a controlled variable.1
• Evaluating comprehensively across 23 classification, segmentation, and low-data tasks under matched tokenization, capacity, and protocol, we find that shifting global attention from the spatial to the feature axis leaves from-scratch performance essentially unchanged, yet markedly strengthens transfer.1

Sources (1)

• [1]Learning Features, Not Layouts: Feature-Axis Attention as a Transfer Prior for Medical Imaging
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 05:16 AM

  Natural-image-pretrained foundation models are increasingly adapted to medical imaging, yet transfer remains unreliable under domain shift, modality heterogeneity, and label scarcity.

  Standard vision transformers place all global modeling on self-attention over spatial tokens, an object-layout prior well suited to natural scenes but not to medical images, where the discriminative signal lies in feature correlations and texture that recur across scanners while layout does not.

Extractive summary: sentences quoted from the sources.