← Tutorials12 / 22
What is an LLM?
Attention has no notion of word order by itself. How does the model know positions?Tap to reveal
Position information is added to the token vectors. The original transformer added fixed sine/cosine patterns; many modern models use RoPE (rotary position embeddings), which rotates queries and keys by an angle that depends on position.
Without it, 'dog bites man' and 'man bites dog' would look the same.