Atom-JEPA: Joint-Embedding Predictive Architecture for 3D Atomistic Systems

Key points

• While deep learning has driven considerable progress in modeling atomistic systems in recent years, self-supervised pretraining in this domain has not yet achieved comparable downstream generalization.1
• We pretrain Atom-JEPA on large-scale molecular and crystalline datasets and evaluate its transfer performance by fine-tuning on a diverse set of downstream property prediction tasks.1
• Atom-JEPA achieves state-of-the-art performance on molecular ADMET and quantum-chemical property prediction tasks, and is highly competitive in predicting the physical properties of crystalline materials.1
• These results demonstrate the potential of latent-space predictive pretraining to support broad downstream generalization from structural data alone.1

Sources (1)

• [1]Atom-JEPA: Joint-Embedding Predictive Architecture for 3D Atomistic Systems
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 02:13 PM

  To address this, we introduce Atom-JEPA, a self-supervised pretraining framework that learns latent representations from unlabeled 3D structures through complementary atom-level and substructure-level objectives inspired by joint-embedding predictive architectures.

  While deep learning has driven considerable progress in modeling atomistic systems in recent years, self-supervised pretraining in this domain has not yet achieved comparable downstream generalization.

Extractive summary: sentences quoted from the sources.