GraphDecisionBench: Benchmarking System One Models on Graph Tasks

Key points

• Large language models (LLMs) are increasingly explored for graph understanding and decision-making, while System One models such as Jev select directly from supplied options.1
• However, the capabilities of System One models on graph-related tasks remain unclear.1
• We evaluate Jev and related choice-based models alongside language-model baselines, covering fourteen model-interface configurations.1
• Jev's results illustrate the benchmark's central distinctions: accurate adjacency recognition does not guarantee broader structural correctness, joint graph-text input does not consistently improve prediction, and feasible construction does not establish high solution quality.1

Sources (1)

• [1]GraphDecisionBench: Benchmarking System One Models on Graph Tasks
  Hugging Face Daily Papers · Oct 9, 12:00 AM

  We introduce GraphDecide, a model-independent benchmark that combines structural task profiles, matched graph-text input contrasts and heuristic-proposal controls to diagnose graph decision performance.

  Large language models (LLMs) are increasingly explored for graph understanding and decision-making, while System One models such as Jev select directly from supplied options.

Extractive summary: sentences quoted from the sources.