TreeHCA: Hindsight-Guided Exploration and Credit Assignment for Multi-Turn LLM Agents

Key points

• Outcome-supervised group-relative policy optimization (GRPO) assigns the same advantage to every token in an LLM agent trajectory, leaving useful and unhelpful decisions undifferentiated.1
• Hindsight credit assignment (HCA) can refine this feedback, but its estimates rely on trajectories containing informative alternatives at consequential histories.1
• Because this exploration changes the distribution of sampled continuations, we derive the corresponding inverse-hindsight importance correction, including an exact finite-pool identity.1
• Across seven question-answering benchmarks and three Qwen backbones, TreeHCA outperforms all baselines in aggregate exact match, validating the effectiveness of paired exploration and credit assignment for multi-turn LLM agent training.1

Sources (1)

• [1]TreeHCA: Hindsight-Guided Exploration and Credit Assignment for Multi-Turn LLM Agents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 12:23 AM

  We introduce TreeHCA, which couples hindsight-guided exploration with hindsight credit assignment.

  Outcome-supervised group-relative policy optimization (GRPO) assigns the same advantage to every token in an LLM agent trajectory, leaving useful and unhelpful decisions undifferentiated.

Extractive summary: sentences quoted from the sources.