Back in Style: A Sociolinguistic Approach to Authoring and Measuring Persona Fidelity in User Simulation
In this pilot study, we ask whether fidelity can instead be measured deterministically by treating a user persona sociolinguistically: as a social type that emerges from observable linguistic style, rather than one predicted by labels or descriptions a model must extrapolate into behaviour.
ProofPaper ↗
Key points
- As agentic systems gain commercial popularity, user simulators increasingly serve as measurement instrument for their evaluation.
- We author personas as concrete stylistic rates, which lets us transfer two established, model-free instruments -- authorship-verification stylometry and lexicon-based content analysis -- as fidelity diagnostics.
- We A/B-test the sociolinguistic schema against a flat descriptive baseline across five task-oriented customer-service agents.
- We argue that a sociolinguistic approach to persona design is a promising path towards more diverse and representative user personas, and that these metrics are most valuable in an error-attribution analysis, localizing where fidelity breaks down.
Sources (1)
- [1]Back in Style: A Sociolinguistic Approach to Authoring and Measuring Persona Fidelity in User SimulationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 11:21 PM
In this pilot study, we ask whether fidelity can instead be measured deterministically by treating a user persona sociolinguistically: as a social type that emerges from observable linguistic style, rather than one predicted by labels or descriptions a model must extrapolate into behaviour.
As agentic systems gain commercial popularity, user simulators increasingly serve as measurement instrument for their evaluation.
Extractive summary: sentences quoted from the sources.