Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling
In this work, we introduce Hierarchical Continuous Diffusion Language Models (H-CDLMs), a simple framework that further improves continuous DLMs with minimal compute and parameter overhead.
Key points
- Diffusion Language Models (DLMs) hold the promise of order-agnostic, parallel text generation.
- Recently, continuous diffusion and flow matching models have seen substantial gains, driven by carefully crafted token representations and diffusion/flow spaces.
- Drawing on the discrete DLM and continuous image diffusion literature on joint diffusion, we diffuse multiple modalities in parallel.
- We propose a general setup that allows per-modality samplers and schedules to enhance the interplay between modalities.
Sources (1)
- [1]Denoising Hierarchical Representations: Joint Continuous Diffusion for Language ModelingarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 05:38 PM
In this work, we introduce Hierarchical Continuous Diffusion Language Models (H-CDLMs), a simple framework that further improves continuous DLMs with minimal compute and parameter overhead.
Diffusion Language Models (DLMs) hold the promise of order-agnostic, parallel text generation.
Extractive summary: sentences quoted from the sources.