The Lattice of Transition Laws
Diffusion and autoregression (AR) have long been seen as different categories of generative models, with diffusion specialising in continuous fields and AR specialising in discrete tokens.
Key points
- In this paper, we ask whether the performance of decoding schedules of one model can be predicted before decoding at a fixed number of steps.
- We describe diffusion, AR, and models in between as paths on one corruption lattice, and define the cost of a schedule as the dependence its parallel steps discard.
- In particular, for data that are Markov on a graph and dependent along its paths, the fewest steps equal the graph's treedepth, which is logarithmic in the length of a sequence and linear in the side length of a grid.
- This work therefore provides a design principle for decoding for future AR models, diffusion models, and anything in between.
Sources (2)
- [1]The Lattice of Transition LawsHugging Face Daily Papers · Oct 8, 12:00 AM
Diffusion and autoregression (AR) have long been seen as different categories of generative models, with diffusion specialising in continuous fields and AR specialising in discrete tokens.
In this paper, we ask whether the performance of decoding schedules of one model can be predicted before decoding at a fixed number of steps.
- [2]The Lattice of Transition LawsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 04:13 AM · same content
Extractive summary: sentences quoted from the sources.