← Tutorials10 / 22
What is an LLM?
Why can a GPT-style model only attend to earlier tokens?Tap to reveal
The cat sat on The ● · · · cat ● ● · · sat ● ● ● · on ● ● ● ● ● can attend · masked
It's trained to predict the next token. If a position could see the future, it would just copy the answer and learn nothing.
A causal mask blocks attention to later positions. That lets one training pass learn from every position of a text at once.