RoboQuest: Generalist Physical Agents that Search, Inspect and Test
We thus introduce RoboQuest, a benchmark for goal-directed embodied exploration, where agents must actively acquire task-relevant information through physical interaction, use the resulting evidence to adapt subsequent actions, and autonomously decide when to commit to task completion.
ProofPaper ↗
Key points
- Recent advances in multimodal foundation models have made them capable generalist physical agents for a range of manipulation tasks.
- However, successful operation in an unfamiliar environment may require an agent to seek task-relevant information through interaction when it is absent from the observations: it may need to determine where a relevant object is, inspect an unobserved property, or discover the effect of an unfamiliar tool.
- RoboQuest comprises ten mobile manipulation tasks centered on three forms of uncertainty: search, manipulation-based inspection, and interactive testing.
- We evaluate five frontier multimodal agents through a common visuomotor interface, as well as a $π{0.5}$ policy fine-tuned on the full-episode demonstrations we release.
Sources (1)
- [1]RoboQuest: Generalist Physical Agents that Search, Inspect and TestarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 04:46 PM
We thus introduce RoboQuest, a benchmark for goal-directed embodied exploration, where agents must actively acquire task-relevant information through physical interaction, use the resulting evidence to adapt subsequent actions, and autonomously decide when to commit to task completion.
Recent advances in multimodal foundation models have made them capable generalist physical agents for a range of manipulation tasks.
Extractive summary: sentences quoted from the sources.