Bridging Human and Model Attention: Explainability Analysis of CNN, Mamba, and ViT Architectures with Gaze-Based Validation

Key points

• High-risk applications of deep learning require precise characterization of model decision-making.1
• Unlike predominantly qualitative studies, we quantify the alignment between model attention and human visual perception using eye-tracking ground truth.1
• Our results reveal distinct attention dynamics: CNNs tend to exhibit a more global focus, with spatially more diffuse saliency maps that correlate well with human perception in general contexts, while ViTs demonstrate fine-grained, patch-level attention, often focused on discriminative microfeatures.1
• The Mamba architecture shows biased behavior: it achieves strong explainability in medical tasks but exhibits divergent intensity distributions in natural images; our analytical characterization of the metrics makes this dissociation explicit, since near-unit support overlap is attained simultaneously with a vanishing linear correlation.1

Sources (1)

• [1]Bridging Human and Model Attention: Explainability Analysis of CNN, Mamba, and ViT Architectures with Gaze-Based Validation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 08:33 PM

  We conduct a systematic explainability evaluation across Convolutional Neural Networks (CNNs), Vision Transformers (ViTs), and Vision Mamba (ViM).

  High-risk applications of deep learning require precise characterization of model decision-making.

Extractive summary: sentences quoted from the sources.