Stationary Bias and Extrapolation in Nonlinear Two-Timescale Stochastic Approximation

Key points

• Constant-step stochastic approximation generally has a nonzero stationary mean error that persists under time averaging.1
• Under stated smoothness assumptions and conditions on the stationary distribution, we derive a first-order bias expansion whose error bound remains uniform as the slow step size becomes much smaller than the fast step size.1
• We verify localization for temporal-difference learning and compare finite-run extrapolation at equal update budgets.1
• For finite runs, we bound the initialization error of tail averages on both timescales under an additional coupling assumption.1

Sources (1)

• [1]Stationary Bias and Extrapolation in Nonlinear Two-Timescale Stochastic Approximation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 03:28 PM

  This paper studies that error for nonlinear two-timescale recursions driven by an exogenous finite-state Markov chain.

  Constant-step stochastic approximation generally has a nonzero stationary mean error that persists under time averaging.

Extractive summary: sentences quoted from the sources.