← Tutorials14 / 22
What is an LLM?
How is an LLM trained (pretraining)?Tap to reveal
text ─▶ model ─▶ predicted probs
│ compare with
│ true next token
▼
loss (cross-entropy)
│ backprop
▼
update all weights1. Take a huge amount of text and cut it into token sequences. 2. At every position, the model predicts the next token. 3. Measure the error with cross-entropy loss. 4. Backpropagation computes how each parameter should change; an optimizer (AdamW) nudges them. 5. Repeat for trillions of tokens.
It's 'self-supervised': the text is its own answer key, so no human labels are needed.