UNITAS: A 3D-Native World Action Model for Embodied Manipulation

Key points

• World action models (WAMs) aim to answer a coupled physical question: given a task instruction, what motion should the robot execute, and how will that motion change the surrounding world?1
• Action flow represents human hands and robot grippers as 3D point trajectories, while scene flow describes scene-point displacements conditioned on these trajectories.1
• World-aligned 3D positional embeddings ground visual tokens with or without depth input, and a physical-time trajectory tokenizer encodes each point trajectory as one token anchored at its current 3D position.1
• This interface supports both direct action execution and action-conditioned scene prediction.1

Sources (1)

• [1]UNITAS: A 3D-Native World Action Model for Embodied Manipulation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 03:02 PM

  We introduce UNITAS, to our knowledge the first 3D-native world action model that unifies observations, actions, and scene dynamics in a shared metric 3D frame within each interaction, using a common representation across robot embodiments and human hands.

  World action models (WAMs) aim to answer a coupled physical question: given a task instruction, what motion should the robot execute, and how will that motion change the surrounding world?

Extractive summary: sentences quoted from the sources.