A Critical Look at Power Sampling when Test Time Sharpening LLMs

Key points

• Recent work therefore seeks RLVR's gains without needing reward signals, by sampling from the power distribution $πα\propto π{base}^α$ via Markov chain Monte Carlo or sequential Monte Carlo.1
• We show that these samplers fail to converge to $πα$ under realistic compute budgets, and that their gains are largely recreated via simple Best-of-$N$ with appropriate grading .1
• This suggests they succeed by selecting high-probability sequences, not by approximating $πα$, raising a natural question: is the power distribution itself a worthwhile target?1
• Together, our results suggest that the value of power sampling lies not in faithfully targeting $πα$, but in the simpler goal of extracting sequences the base model already deems likely.1

Sources (1)

• [1]A Critical Look at Power Sampling when Test Time Sharpening LLMs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 02:29 PM

  Reinforcement learning with verifiable rewards (RLVR) has substantially improved LLM reasoning, yet growing evidence suggests it mainly performs distribution sharpening: concentrating probability mass on reasoning paths the base model already produces.

  Recent work therefore seeks RLVR's gains without needing reward signals, by sampling from the power distribution $π_α\propto π_{base}^α$ via Markov chain Monte Carlo or sequential Monte Carlo.

Extractive summary: sentences quoted from the sources.