What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied Agents

Key points

• Memory allows large language model (LLM) agents to use past experience, often stored as textual records, when responding to requests.1
• We introduce MeMoREval, a benchmark for studying this problem through a controlled experience-request protocol.1
• Our reference evaluations reveal that storing frames outperforms storing generated descriptions at lower write-time inference cost for the primary model, while storing frames with scene graphs achieves the highest aggregate task success for every tested model.1
• Extensions to accumulated experience, selective frame retention, and visual input to memory writers demonstrate how the benchmark accommodates further memory-design investigations.1

Sources (1)

• [1]What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied Agents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 06:00 PM

  For embodied agents that control robots, visual and spatial observations introduce a broader choice of memory modalities, affecting both the information available for subsequent tasks and the model inference costs of constructing and using memory.

  Memory allows large language model (LLM) agents to use past experience, often stored as textual records, when responding to requests.

Extractive summary: sentences quoted from the sources.