Skip to content

MDLM

Generate text with masked diffusion

Visit official site

What MDLM is

The model is described as a masked diffusion language model that generates text through masking and diffusion, with improved training and a simplified objective.

What you can do with it

  • Generate text through ancestral sampling
  • Produce text of arbitrary length
  • Train an encoder-only language model

Model notes

  • Training uses a weighted-average masked cross-entropy loss
  • The objective corresponds to a principled variational lower bound
  • The material reports low perplexity on One Billion Words

Getting started

  • Learn the basic principle and functions
  • Obtain the model and training code
  • Prepare masked and unmasked text samples

Pros & cons

✓ What we like

  • An efficient sampler comes with an encoder-only setup
  • Long-range dependencies and controllable generation are named strengths

! What to watch out for

  • Training needs masked and unmasked text samples
  • The objective is tied to a variational lower bound

FAQ

How is it trained?

With a weighted-average masked cross-entropy loss and a simplified objective.

How is text generated?

The material names ancestral sampling for text generation.

Last reviewed: 2026-11-15

More AI writing assistant tools

View all →

How we review