The Representation-based Masked Diffusion Model (RMDM) is a new approach to language modeling using masked diffusion models. Existing masked diffusion models often update multiple masked tokens independently, which can result in incoherent outputs. The RMDM framework addresses this limitation by incorporating a text representation to guide the parallel updates. Specifically, the model encodes text into a continuous semantic space using a pretrained encoder. An invertible transformation then normalizes the representation distribution to a Gaussian prior. This facilitates efficient sampling during generation. The masked diffusion model is trained to learn the conditional text distribution, using the latent semantic representation as global guidance. Empirical results show that RMDM significantly improves generation quality, particularly in aggressive few-step sampling regimes. The model’s design allows for more precise coordination of token updates, leading to more coherent and faithful text generation. Source: https://arxiv.org/abs/2609.12382
Loading