CM-DPO: Constraint-Margin Direct Preference Optimization for LLM Planning

Key points

• Direct Preference Optimization (DPO) treats all constraint violations equally: a $1 budget overshoot and a $1,000 overshoot induce the same training signal.1
• It is also susceptible to length and style bias when preference pairs come from different model families.1
• To supply CM-DPO with bias-reduced training pairs, we generate preference data through procedurally generated constraint profiles (DCCG) and minimal-edit distillation from a reasoning teacher (RT-MED), within a framework we call SynPlan-R.1
• On TravelPlanner, NaturalPlan, and out-of-distribution PlanBench, an 8B model fine-tuned with CM-DPO achieves 89.2% pass rate and 93.4% solve rate, matching multi-agent systems at 13x lower latency while outperforming GPT-4o on unseen Blocksworld by 9.2 points.1

Sources (1)

• [1]CM-DPO: Constraint-Margin Direct Preference Optimization for LLM Planning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 11:30 PM

  We introduce Constraint-Margin DPO (CM-DPO), which replaces DPO's binary preference signal with a continuous margin derived from a deterministic symbolic verifier and scaled by violation severity.

  Direct Preference Optimization (DPO) treats all constraint violations equally: a $1 budget overshoot and a $1,000 overshoot induce the same training signal.

Extractive summary: sentences quoted from the sources.