Evaluating LLM Creativity through Long-Tail Performance

Key points

• We propose that this shared target can be formalized as task-specific performance in the long tail of a reference input-output distribution, which describes how common input-output pairs are relative to a chosen source of text.1
• We ground this proposition in the novelty-quality product formulation and evaluate it empirically across 15 benchmarks and 26 open-weight models, using n-gram absence from the pretraining data as a novelty proxy.1
• Moreover, on both creativity and standard benchmarks, model rankings from higher-tail subsets of each benchmark align more closely with human creativity judgments in LM-Arena.1
• Building on these findings, we introduce q@Tail($α$), a lower-cost creativity diagnostic that measures mean task quality on the tail fraction of an existing benchmark.1

Sources (1)

• [1]Evaluating LLM Creativity through Long-Tail Performance
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 05:03 AM

  Creativity benchmarks for large language models (LLMs) vary widely in task and metric design.

  We propose that this shared target can be formalized as task-specific performance in the long tail of a reference input-output distribution, which describes how common input-output pairs are relative to a chosen source of text.

Extractive summary: sentences quoted from the sources.