On the Design Space of Continuous Diffusion Language Models

Key points

• Self-conditioning also aids in recognising tokens dominated by large-norm Gaussians$\unicode{x2013}$an issue known as token identifiability.1
• To address it directly, we introduce discrete reveals to $DoubleCat$, yielding $SemiCat$, which achieves discrete diffusion-level performance on TinyGSM, reaching $49.94%$ with standard training hyperparameters and a tuned sampler.1
• At the $1.7$B scale, $SemiCat$ outperforms discrete diffusion on four of six likelihood-based evaluations, and surpasses one-hot flows on unconditional generation.1
• On embedded spaces, we find that cross-entropy drives embeddings towards a geometry whose decoding error rate closely matches the one-hot one.1

Sources (1)

• [1]On the Design Space of Continuous Diffusion Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 04:41 PM

  Continuous diffusion language models (CDLMs) promise to bring the benefits of continuous flow modelling to language, including accelerated sampling and controllable generation.

  Self-conditioning also aids in recognising tokens dominated by large-norm Gaussians$\unicode{x2013}$an issue known as token identifiability.

Extractive summary: sentences quoted from the sources.