Decoupled Optimization for Teacher-Student Semi-Supervised Learning via a Pioneer Student

Key points

• Despite their effectiveness, we find both mechanisms introduce distinct optimization pathologies.1
• First, parameter coupling enforces strict synchronization between teacher and student, where strong regularization on the student degrades the teacher's fitting ability, thereby limiting the permissible generalization intensity.1
• Second, the imbalance in gradient update consistency between labeled and unlabeled losses drives the shared parameters to prematurely converge to labeled-dominated local minima, creating a bottleneck for global optimization.1
• To address both issues, we propose the Pioneer Student (PiS), an auxiliary branch that operates in an independent parameter space and periodically transfers accumulated knowledge back to the T-S model.1

Sources (1)

• [1]Decoupled Optimization for Teacher-Student Semi-Supervised Learning via a Pioneer Student
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 07:54 AM

  Semi-supervised learning (SSL) relies on two core mechanisms: self-training under the Teacher-Student (T-S) framework and joint optimization of labeled and unlabeled losses.

  Despite their effectiveness, we find both mechanisms introduce distinct optimization pathologies.

Extractive summary: sentences quoted from the sources.