Improving Semantic Coverage in AI Test & Evaluation through Generative Augmentation

Key points

• In test and evaluation (T&E) of AI models, evaluators typically have access only to a held-out test dataset and no knowledge of the training data or its semantic composition.1
• First, vision-language models (VLMs) perform zero-shot binary labeling of semantic factors, and an embedding-space clustering step identifies which semantics most affect downstream model behavior.1
• Second, diffusion-based image-to-image models transform existing covered images into the underrepresented semantic domain, preserving scene content while augmenting high-level appearance.1
• We validate the full pipeline on both image classification and object detection under day/night semantic gaps, showing that the augmented test datasets better distinguish models with and without exposure to the missing semantic, thereby yielding T&E conclusions that more closely match those of an ideal, fully covered dataset.1

Sources (1)

• [1]Improving Semantic Coverage in AI Test & Evaluation through Generative Augmentation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 02:57 PM

  We propose a three-stage generative AI pipeline to detect and fill such semantic gaps without requiring ground-truth labels of these semantics.

  In test and evaluation (T&E) of AI models, evaluators typically have access only to a held-out test dataset and no knowledge of the training data or its semantic composition.

Extractive summary: sentences quoted from the sources.