RESCUE: Is This Alarm Worth Acting On? Counterfactual Evaluation of VLA Runtime Monitors

Key points

• Runtime failure monitors for robot policies are typically evaluated by how accurately they distinguish failed from successful executions.1
• By restoring simulator and policy states at execution checkpoints, RESCUE compares alternative actions, identifies the latest checkpoint satisfying a specified recovery criterion, and evaluates whether acting on alarms reduces expected task-completion cost.1
• We evaluate RESCUE with OpenVLA-OFT and $π{0.5}$ over LIBERO-Spatial and LIBERO-Plus using 128,340 counterfactual branches.1
• Recoverability depends on the state--intervention pair rather than on robot state alone: at $τ=0.5$, 14 of 66 OpenVLA-OFT/LIBERO-Plus failures contain a recoverable checkpoint under continuation, compared with 34 of 66 under continuation or scripted recovery, while recovery reduces recoverability in other conditions.1

Sources (1)

• [1]RESCUE: Is This Alarm Worth Acting On? Counterfactual Evaluation of VLA Runtime Monitors
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 12:38 PM

  We introduce RESCUE, a counterfactual framework for evaluating whether a monitor's alarms enable useful interventions.

  Runtime failure monitors for robot policies are typically evaluated by how accurately they distinguish failed from successful executions.

Extractive summary: sentences quoted from the sources.