When KL Regularization Misfires in Group Policy Optimization

Key points

• This motivates studying how reference-policy information should enter group-relative updates.1
• We analyze seven potential failure modes in the interactions between KL and rewards: residual KL updates after reward clipping, after gradient cancellation, and in groups with identical rewards; KL growth with response length and an imbalance in its relative contribution; KL concentration on a small number of tokens; and sampling noise when k1 is incorporated into rewards.1
• We propose Zero-Sum Calibrated Policy Optimization (ZCPO), which uses relative drift measured by conditional KL to calibrate within-group reward coefficients and integrates them into the base surrogate.1
• Mathematical reasoning experiments and ablations support this design's effectiveness in our settings.1

Sources (1)

• [1]When KL Regularization Misfires in Group Policy Optimization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 03:39 PM

  Why does removing reference-policy KL regularization sometimes improve group policy optimization?

  This motivates studying how reference-policy information should enter group-relative updates.

Extractive summary: sentences quoted from the sources.