Neural networks are more modular than single neurons suggest

Key points

• Neural networks routinely develop representations in which individual neurons contribute to multiple distinct functions, leading to the widespread view that such networks lack modular organization.1
• We develop a unified framework for learning decompositions at each level and evaluating them causally through selective ablation.1
• Across vision models, language models and multi-task Recurrent Neural Networks (RNNs), we find that networks which lack unit-level modularity nonetheless often exhibit clear functional modularity at the subspace level.1
• The absence of unit-level modularity frequently coexists with clear subspace-level modularity across domains, enabling targeted intervention on specific computations even in networks where no individual neuron is specialized, with broad implications for mechanistic interpretability and for understanding functional organization in both biological and artificial neural systems.1

Sources (1)

• [1]Neural networks are more modular than single neurons suggest
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 12:58 AM

  We challenge this conclusion by introducing a formally nested hierarchy of modularity: unit-level, which requires distinct computations to occupy disjoint neurons; orthogonal, which asks whether a rotation reveals functionally independent subspaces; and linear, which asks whether a bounded linear map can separate task-relevant computations into independent subspaces.

  Neural networks routinely develop representations in which individual neurons contribute to multiple distinct functions, leading to the widespread view that such networks lack modular organization.

Extractive summary: sentences quoted from the sources.