Secure-CUA: Controlling Untrusted Influence in Computer-Use Agents

Key points

• An adversary controlling this untrusted content can embed instructions or misleading visual cues to change the agent's intended action or redirect its commands to the wrong interface target.1
• In an ideal execution model, we show that enforcing both requirements at each step protects execution traces.1
• We instantiate this model in Secure-CUA, our system for secure CUA execution.1
• We evaluate Secure-CUA under benign conditions on 400 WebArena tasks using three frontier models across $5$ seeds, yielding $6,000$ execution traces.1

Sources (1)

• [1]Secure-CUA: Controlling Untrusted Influence in Computer-Use Agents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 05:24 AM

  Computer-use agents (CUAs) perform tasks across applications (such as desktops, mobile apps, and web browsers) by observing graphical interfaces and issuing commands such as clicks and keystrokes.

  An adversary controlling this untrusted content can embed instructions or misleading visual cues to change the agent's intended action or redirect its commands to the wrong interface target.

Extractive summary: sentences quoted from the sources.