ConClick: Element-Wise Contrastive Learning Improves GUI Grounding

Key points

• GUI grounding, the task of locating the interface element specified by a user instruction, is a fundamental capability of GUI agents.1
• We systematically evaluate 10 grounding models across five datasets and identify a major error pattern: most incorrect clicks hit non-target interface elements rather than the background, with different models repeatedly selecting the same distractors.1
• ConClick improves the grounding accuracy of Qwen3-VL-8B from 54.71% to 66.03% on ScreenSpot-Pro, from 83.72% to 86.28% on MMBench-GUI L2, and from 63.33% to 70.20% on OSWorld-G, outperforming prior methods built on the same or similarly sized backbones.1
• We further introduce ReClick, an inference-time strategy that revisits promising regions and compares predictions across competing candidates, yielding an additional 2.96 percentage-point improvement in average accuracy across the three benchmarks.1

Sources (1)

• [1]ConClick: Element-Wise Contrastive Learning Improves GUI Grounding
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 09:20 AM

  Motivated by these findings, we propose ConClick, a contrastive learning framework that teaches models to distinguish the target element from plausible distractors.

  GUI grounding, the task of locating the interface element specified by a user instruction, is a fundamental capability of GUI agents.

Extractive summary: sentences quoted from the sources.