Autoregressive generation, also known as next-token prediction, is the mechanism AI tools use to generate text-based output. If you’ve ever sent a prompt to Claude or ChatGPT, you’ve experienced autoregressive text generation firsthand.
In simple terms, autoregressive generation is the act of building sentences one word at a time, sequentially, based on the words that have already been selected.
How does autoregression work?
When generating output, a large language model (LLM) will examine the prompt it is given and start with a single word, perhaps “Enemies.”
It then builds a word cloud of things that might make sense after the word “Enemies”. Each word in the cloud has a probability score associated with its applicability given the prompt, or context.
- surrender?
- quit?
- retreat?
- run?
- yield?
- apologize?
All may be valid words, and the LLM gets to choose which one to insert next.
Claude, Gemini & ChatGPT build probability clouds of words that would make the most sense inserted into a sentence.
What is a probabilistic word cloud?
You can think of the words closest to the center of the word cloud as having the highest applicability, and the ones further away having less. All words in the cloud are valid, and the LLM may choose any one of them as the next word in the sentence.
After the second word is added to the first, a new word cloud is created based on the context created by the first two words, and the new word is appended.
Words are added to a sentence, one token at a time, automatically based on the words previously chosen.
Autoregressive generation is the manner in which ChatGPT generates text-based output.
Why is it called autoregression?
The term “regress” or “regression” means going back to a previous state, or in this case, looking at a set of previously generated values. The fact that the LLM chooses new words based on the words it previously choose is why it’s called “regressive.”
Add on the fact that this happens automatically, and you get the first part of the term being defined here: autoregression. The fact that this process is used to create new content is what makes it generative.
Thus the term: autoregressive generation.
What is next-token prediction?
To be more accurate, LLMs don’t predict words, they predict “tokens”, which may be any small chunk of text, from one long word, to punctuation, to a collection of smaller words.
That’s why the process is also called next-token prediction or autoregressive token generation. But regardless of what you call it, the song remains the same. Content gets regressively generated one little piece at a time.
Now autoregressive generation isn’t the only game in town when it comes to generating text. Diffusion-based text generation is an alternate strategy where entire sentences are built at once, and words magically manifest into the appropriate spots in the text.
What’s better, autoregression or text diffusion?
One problem with autoregressive generation is that after 6 or 7 words, the LLM is locked into a given thought process and can’t go back. Once a token has been emitted, the model won’t go back and replace it.
Text diffusion gives a bit more flexibility, as the beginning of a sentence can be formed after knowing the words towards the end, making it more flexible, although today it remains far less common.
But that’s how the ChatGPT and Claude Code sausage is made: one autoregressive token at a time, and from my perspective, it works pretty darn good.