What Should SFT Learn? Conditional Information Gain for Generalizable Post-Training

Key points

• Yet forcing a model to imitate externally provided trajectories may also interfere with behaviors already supported by pretraining, limiting generalization beyond the training distribution.1
• Recent dynamic SFT methods mitigate this issue by reweighting token-level supervision according to model-side signals such as predictive probability, prior support, or distributional compatibility.1
• We propose CIG-SFT, an information-theoretic framework that estimates the task-conditioned information carried by each demonstrated token.1
• Under teacher forcing, we compare the token likelihood from a frozen reference model with and without access to the task input while keeping the gold solution prefix fixed.1

Sources (1)

• [1]What Should SFT Learn? Conditional Information Gain for Generalizable Post-Training
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 04:57 AM

  Supervised fine-tuning (SFT) remains a widely used approach for post-training large language models, because fixed demonstrations can directly introduce desired reasoning strategies and behavioral patterns.

  Yet forcing a model to imitate externally provided trajectories may also interfere with behaviors already supported by pretraining, limiting generalization beyond the training distribution.

Extractive summary: sentences quoted from the sources.