← Tutorials8 / 22
What is an LLM?
What does self-attention do?Tap to reveal
The cat didn't eat because it
▲ │
└──── strong attention ───┘It lets each token look at the tokens before it and pull in information from the relevant ones.
In 'The cat didn't eat because it was full', the vector for 'it' attends strongly to 'cat', so the model knows what 'it' refers to.