Text diffusion is a method AI tools use to generate text by repeatedly refining a sequence of words or tokens until a coherent sentence or piece of text emerges.
Unlike autoregressive generation, where an LLM sequentially builds texts one token at a time, text diffusion has the ability to figure out many parts of a sentence at once. That’s its superpower.
How does text diffusion work?
Imagine you ask an AI model to generate a sentence about enemies retreating from battle.
A diffusion model might initially represent the answer as something like this:
???? ???? ???? ???? ???? ????
At first, the model doesn’t know exactly which words belong in each position.
It then makes a pass over the entire sequence and starts filling in words that seem appropriate:
Enemies ???? ???? from ???? ????
Then it makes another pass:
Enemies will ???? from the ????
Then another:
Enemies will retreat from the battlefield.
Each pass makes the sentence a little less uncertain and a little more coherent.
The important part is that the sentence doesn’t necessarily get constructed one word at a time.
Words can appear in different places during the same generation process. They can also be reconsidered and replaced as the model develops a better understanding of what the final sentence should say.
Why is it called diffusion?
The term “diffusion” comes from the family of diffusion models that became well known for generating images.
A diffusion model can be trained by taking useful information and progressively corrupting it with noise.
Imagine taking a perfectly clear photograph and adding more and more static until the original image is almost impossible to see.
The model learns how to reverse that process.
-
Give it noise.
-
Remove some uncertainty.
-
Remove some more.
-
Keep refining.
Eventually, something meaningful emerges.
With image diffusion, noisy pixels become recognizable images.
With text diffusion, uncertain or masked tokens of text become recognizable sentences.
Thus the term text diffusion.
What is parallel text generation?
The ability to work on multiple parts of the output is one of the most compelling characteristics of text diffusion.
With autoregressive generation, token number five generally can’t be generated until tokens one through four already exist.
With text diffusion, different parts of the problem can potentially be worked on at the same time.
The exact amount of parallelism depends on the particular diffusion architecture, but the model isn’t inherently restricted to the same strict left-to-right generation process used by an autoregressive LLM.
What’s better, text diffusion or autoregression?
Autoregressive generation is exceptionally good at generating language, and it remains the dominant technique behind many of the LLMs people use today.
Text diffusion simply approaches the problem from a different direction.
Autoregression says:
Choose a token. Then choose another. Then another.
Text diffusion says:
Start with uncertainty. Refine everything. Keep refining until the words fall into place.
Different sausage-making process. Same goal. Just turn a prompt into useful text.