Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?

Key points

• Evaluating this ability is therefore important for understanding how effectively agents acquire and use new knowledge.1
• To address this, we introduce Learn2Play Bench, a benchmark of newly designed text-based games, whose rules are novel or counterintuitive, requiring agents to acquire knowledge through interaction rather than rely solely on pretrained knowledge.1
• Therefore, we evaluate how backbone models, self-evolving methods, and agent harnesses affect agents' learning ability, revealing three findings: (1) Experience retention: Retaining complete records of actions and feedback can support more effective learning than summarizing these experiences into rules or strategies.1
• (2) Human agent gap: Top-performing human players achieve higher peak scores than the evaluated agents.1

Sources (2)

• [1]Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?
  Hugging Face Daily Papers · Oct 8, 12:00 AM

  Learning from experience is essential for LLM agents to adapt to unfamiliar and dynmaic environments.

  Evaluating this ability is therefore important for understanding how effectively agents acquire and use new knowledge.

• [2]Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 12:08 PM · same content

Extractive summary: sentences quoted from the sources.