WASD: Wasserstein-based Knowledge Distillation for Large Language Models

Key points

• Autoregressive large language models (LLMs) have rapidly advanced in capability, but their increasing scale comes with substantial computational and memory costs at inference time.1
• Knowledge distillation (KD) offers a practical solution by transferring knowledge from a large teacher model to a smaller student model via alignment of discrete probability distributions.1
• To ensure computational tractability, we adopt the Sinkhorn divergence and derive a gradient-equivalent objective that can be efficiently optimized without introducing additional networks.1
• Our results highlight the importance of semantic information encoded in the token space for effective distribution alignment in LLM distillation.1

Sources (1)

• [1]WASD: Wasserstein-based Knowledge Distillation for Large Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 03:58 AM

  We propose Wasserstein-based knowledge distillation (WASD) for LLMs, which incorporates token-level semantic information via the Wasserstein-based distance with a cost matrix derived from token embeddings.

  Autoregressive large language models (LLMs) have rapidly advanced in capability, but their increasing scale comes with substantial computational and memory costs at inference time.

Extractive summary: sentences quoted from the sources.