Neural expectation-maximization and meta-models

Key points

• In this view, a Transformer performs inference in partially observed Markov Random Fields (MRFs) whose structure and parameters are not given a priori but are inferred from data.1
• We show that self-attention implements a differentiable approximation to max-product LBP on an input-conditioned, dynamically structured MRF, with attention weights encoding learned structural affinities.1
• We further establish an EM correspondence at the level of depth: each layer updates latent representations in an E-step-like forward pass, while the transition $θ^{(l)} \to θ^{(l+1)}$ across layers acts as an implicit M-step.1
• Building on this interpretation, we introduce meta-models, a design paradigm in which inputs generate instance-specific parameters, unifying mechanisms such as self-attention, mixture-of-experts, and gated units within a single framework.1

Sources (1)

• [1]Neural expectation-maximization and meta-models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 09:37 PM

  We interpret Transformer networks as learned approximations to a hybrid probabilistic-inference procedure that combines max-product Loopy Belief Propagation (LBP) with Expectation--Maximization (EM).

  In this view, a Transformer performs inference in partially observed Markov Random Fields (MRFs) whose structure and parameters are not given a priori but are inferred from data.

Extractive summary: sentences quoted from the sources.