When Interfaces Speak: Data-Aware Generative UI Harness for Active Interaction

Key points

• Most human-agent interaction today remains text-based.1
• Natural language can impose cognitive overload, ambiguity, information chaos, and slow input for complex tasks; ephemeral generative UIs can present structured information and guide users toward task completion.1
• Training the coder with reinforcement learning is challenging: verifiable rewards for interactive UI generation require costly execution, while LLM-as-a-Judge rewards are prone to reward hacking.1
• We introduce UI-TAU Bench, a benchmark for active human-agent interaction through generated UI code, built on 10 real-world domain databases constructed from public data sources and based on Tau-Bench tool-use settings, with Lite (300 tasks) and Full (1,000 tasks) splits.1

Sources (1)

• [1]When Interfaces Speak: Data-Aware Generative UI Harness for Active Interaction
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 02:49 AM

  We propose GenUI-Harness, a multi-agent harness pairing a Tool Agent for information retrieval and task execution with a GUI Coder Agent that identifies ambiguities and generates front-end code for structured interfaces.

  Most human-agent interaction today remains text-based.

Extractive summary: sentences quoted from the sources.