SplitJEPA: Learning Invariant and Variant Latent Worlds without Reconstruction

Key points

• Understanding a dynamical world calls for more than a latent state that summarizes its observations: the state should also be organized into the factors that stay shared across related observations and the factors that vary between them.1
• Joint embedding predictive architectures (JEPAs) model the latent state directly and never reconstruct, yet no existing result recovers the invariant and variant parts of the state they learn.1
• How to learn the invariant-variant structure of the latent world without paying for its reconstruction therefore remains open.1
• We prove that, under stationary Gaussian predictive dynamics and a full-rank variation condition, SplitJEPA identifies the invariant and variant subspaces up to independent block-wise isometries, without introducing an observation decoder.1

Sources (1)

• [1]SplitJEPA: Learning Invariant and Variant Latent Worlds without Reconstruction
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 05:19 PM

  To close this gap, we introduce SplitJEPA, a JEPA that jointly recovers the latent state and its invariant and variant organization directly in representation space, without any reconstruction.

  Understanding a dynamical world calls for more than a latent state that summarizes its observations: the state should also be organized into the factors that stay shared across related observations and the factors that vary between them.

Extractive summary: sentences quoted from the sources.