Instance-anchored interaction evidence: Grounding robot plans in human pointing and handling
We propose instance-anchored interaction evidence (IAE), which registers every object of the final scene to its public identifier, keeps each identity through the video by backward mask propagation, and describes every frame by the geometry between hands, forearms and these instances.
ProofPaper ↗
Key points
- A robot that assists people must often act on what a person has shown rather than said: which of several identical cartons was pointed at, or which box was handled.
- An evidence network trained only from task outcomes scores the instances.
- For pointing tasks, a grammar-constrained dynamic program trained with a structured loss decodes object-destination programs; symbolic programs handle reference disambiguation and, without learning, episodic tasks.
- On 1,255 WatchAct benchmark requests, scored by symbolic execution, IAE reaches 64.2% plan success on implicit-intent tasks against 27.5% for a 32B vision-language model (strict success 49.7% against 15.4%), and 46.4% against 27.0% on restoration, reversal and imitation without task-specific training.
Sources (1)
- [1]Instance-anchored interaction evidence: Grounding robot plans in human pointing and handlingarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 03:37 PM
We propose instance-anchored interaction evidence (IAE), which registers every object of the final scene to its public identifier, keeps each identity through the video by backward mask propagation, and describes every frame by the geometry between hands, forearms and these instances.
A robot that assists people must often act on what a person has shown rather than said: which of several identical cartons was pointed at, or which box was handled.
Extractive summary: sentences quoted from the sources.