How Well Do LLMs Reason with Noisy Evidence? An Active Visual Reasoning Benchmark

Key points

• Real-world reasoning rarely reduces to static question answering: agents must actively gather information from tools and sensors that are often noisy and unreliable.1
• Yet most existing active reasoning benchmarks assume that environmental feedback is trustworthy, or introduce noise without exposing an explicit, calibrated uncertainty signal, leaving open how LLMs should reason when the evidence itself is uncertain.1
• We evaluate multiple LLM reasoners on 1,000 instances spanning perception, chart understanding, and knowledge-intensive reasoning.1
• VisualNoiseQA thus provides a controlled playground to study how different LLMs exploit uncertainty signals for robust reasoning.1

Sources (1)

• [1]How Well Do LLMs Reason with Noisy Evidence? An Active Visual Reasoning Benchmark
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 04:49 AM

  We introduce VisualNoiseQA, a novel benchmark for active reasoning under noisy visual feedback.

  Real-world reasoning rarely reduces to static question answering: agents must actively gather information from tools and sensors that are often noisy and unreliable.

Extractive summary: sentences quoted from the sources.