RELATE: An Evaluation Framework for measuring Relational Orientation of Large Language Models

Key points

• Large language models (LLMs) are increasingly used for emotional support, raising concern that sustained use may draw users away from their real-world relationships.1
• Yet existing evaluations primarily focus on the safety, empathy, or helpfulness of responses, leaving under-examined a relational question: where does the model orient the user for continued support?1
• Grounded in psychological and sociological literature, we formalize a taxonomy of relational orientation and present RELATE, a persona-conditioned framework for measuring inward-facing and outward-scaffolding language at the sentence level in multi-turn dialogues.1
• Under automated evaluation, we find that the proportion of sentences labeled as IF is higher at the sixth assistant turn than at the first, while the proportion labeled as OS is substantially lower for hesitant, indirect simulated users than for explicit, reassurance-seeking users.1

Sources (1)

• [1]RELATE: An Evaluation Framework for measuring Relational Orientation of Large Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 07:10 AM

  To address this question, we introduce relational orientation, a property operationalized through two non-exclusive dimensions: inward-facing (IF) language, which positions the AI as the user's ongoing source of support, and outward-scaffolding (OS) language, which encourages real-world human connection.

  Large language models (LLMs) are increasingly used for emotional support, raising concern that sustained use may draw users away from their real-world relationships.

Extractive summary: sentences quoted from the sources.