Reflex: Visually Grounded Reactive Humanoid Control

Key points

• Humans effortlessly react to the world; they adeptly return a tennis shot, catch a falling cup, step aside from a passing cart, or pass a box tossed during truck unloading.1
• Such visual reactive whole-body control remains challenging for humanoids, which must (1) coordinate whole-body control, (2) react to object dynamics in real time, and (3) ground perception in egocentric vision.1
• First, a state-based policy learns whole-body catching with reinforcement learning.1
• Third, an egocentric RGB-D encoder learns to recover this dynamics representation directly from visual observations while keeping the whole-body controller fixed.1

Sources (2)

• [1]Reflex: Visually Grounded Reactive Humanoid Control
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 10:24 PM

  We present Reflex, a three-stage framework that learns these capabilities stage by stage while connecting them through a compact representation of object dynamics.

  Humans effortlessly react to the world; they adeptly return a tennis shot, catch a falling cup, step aside from a passing cart, or pass a box tossed during truck unloading.

• [2]Reflex: Visually Grounded Reactive Humanoid Control
  Hugging Face Daily Papers · Oct 8, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.