PERSIST: Who-What-When Memory Across Sessions for Full-Duplex Spoken Dialogue

Key points

• Modern voice assistants may be shared by multiple users and should be able to answer questions about earlier conversations such as "When did I originally plan to leave?" or adapt their behavior to individual users based on past interactions.1
• PERSIST structures cross-session histories into readable event records and retrieves them with a 3W joint scoring mechanism that combines semantic content, acoustic speaker identity, and temporal state.1
• For real-time full-duplex interaction, PERSIST further reuses intermediate representations from the dialogue backbone, avoiding query-audio re-encoding and reducing retrieval latency from 578.42 ms to 7.03 ms.1
• We also introduce SpokenTrace, a diagnostic benchmark that factorizes evaluation along memory tasks and speaker-query types, exposing failures in recall, speaker attribution, and temporal-state tracking.1

Sources (1)

• [1]PERSIST: Who-What-When Memory Across Sessions for Full-Duplex Spoken Dialogue
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 04:21 AM

  We present PERSIST, a persistent memory system for multi-session, multi-speaker spoken dialogue that explicitly models Who, What, and When.

  Modern voice assistants may be shared by multiple users and should be able to answer questions about earlier conversations such as "When did I originally plan to leave?" or adapt their behavior to individual users based on past interactions.

Extractive summary: sentences quoted from the sources.