Robust Langevin Sampling under State-Dependent Heavy-Tailed Noise with Applications to LLM Fine-Tuning
Existing convergence guarantees for non-log-concave sampling with stochastic gradient Langevin Monte Carlo (SG-LMC) typically assume globally Lipschitz gradients and uniformly bounded noise variance.
ProofPaper ↗
Key points
- We relax both assumptions through generalized smoothness and our proposed state-dependent heavy-tailed oracle with finite $p$-th moments, $1<p\leq 2$.
- Robust adaptive LMC (RA-LMC) addresses both relaxations through robust estimation and a noise-aware step size.
- For all three methods, we establish non-asymptotic convergence guarantees in Fisher information (FI) without assuming dissipativity or a functional inequality.
- Bayesian fine-tuning experiments with LLMs and vision-language models support our oracle model and show that RA-LMC matches or outperforms state-of-the-art baselines in accuracy and uncertainty quantification.
Sources (1)
- [1]Robust Langevin Sampling under State-Dependent Heavy-Tailed Noise with Applications to LLM Fine-TuningarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 01:46 AM
Existing convergence guarantees for non-log-concave sampling with stochastic gradient Langevin Monte Carlo (SG-LMC) typically assume globally Lipschitz gradients and uniformly bounded noise variance.
We relax both assumptions through generalized smoothness and our proposed state-dependent heavy-tailed oracle with finite $p$-th moments, $1<p\leq 2$.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 9, 2026PreSTO: Predictive Subtree Prefetching for Fast LLM Power Sampling
- Oct 8, 2026Understanding Jev, the new model everyone is talking about
- Oct 8, 2026Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review
- Oct 8, 2026SuperNav: An Agentic Navigation System for Any Task in Any Scene
- Oct 8, 2026VibeEdit: Image Editing with Canvas Instructions
- Oct 8, 2026SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models