Predicting Alignment Generalization with Value Representations

Key points

• LLM developers post-train their models to exhibit prosocial values and behavioral traits, which are enumerated in an alignment target.1
• We find that representations based on model activations when applying values in context significantly outperform methods based on textual descriptions of the values.1
• We then show the applicability of representations that predict alignment generalization toward downstream tasks by using them to measure how similar the values in a multi-value alignment target are, which we find is significantly correlated with model robustness.1
• Finally, we show initial evidence towards a shared, model-independent value space, which we use to develop the first taxonomy of LLM values grounded in empirical generalization dynamics.1

Sources (1)

• [1]Predicting Alignment Generalization with Value Representations
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 05:47 PM

  In this paper, we establish the task of alignment generalization prediction, i.e., predicting how fine-tuning a model to follow a given value changes its behavior across a wide range of held-out values.

  LLM developers post-train their models to exhibit prosocial values and behavioral traits, which are enumerated in an alignment target.

Extractive summary: sentences quoted from the sources.