Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models
We demonstrate the approach using a published water-quality stated preference economic valuation survey (Vossler et al. 2023) administered to six models.
ProofPaper ↗
Key points
- Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values.
- Stated-preference economics has faced this problem for decades.
- We argue that this framework is a general method for evaluating language models, and we set out what each concept means for LLM evaluation.
- In this economic application, the validity tests take the form of predictions from economic theory: demand should slope down, and willingness to pay should respond to the scope of the good and to income.
Sources (1)
- [1]Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language ModelsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 05:51 PM
We demonstrate the approach using a published water-quality stated preference economic valuation survey (Vossler et al. 2023) administered to six models.
Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values.
Extractive summary: sentences quoted from the sources.