Whose Ground Truth? Embracing Ambiguity in Human-Centered AI

Key points

• Conventional machine learning and AI systems are largely developed under the assumption that a single definitive ground truth exists, with variability in human annotations often resolved through aggregation or treated as noise.1
• However, for many human-centered tasks, human interpretation is inherently ambiguous, and multiple interpretations of the same input may be simultaneously reasonable and valid.1
• In this position paper, we call for a shift towards modeling the interpretation space of plausible human judgments, while distinguishing meaningful ambiguity from annotation noise.1
• We argue that this perspective should guide how AI systems are represented, learned, evaluated, deployed, and governed, supporting more human-centered AI that better reflects the diversity of human interpretation.1

Sources (1)

• [1]Whose Ground Truth? Embracing Ambiguity in Human-Centered AI
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 07:05 PM

  As AI systems increasingly interact with people and make decisions about them, understanding human interpretations becomes an important part of developing human-centered AI.

  Conventional machine learning and AI systems are largely developed under the assumption that a single definitive ground truth exists, with variability in human annotations often resolved through aggregation or treated as noise.

Extractive summary: sentences quoted from the sources.