← Tutorials14 / 22

What is an LLM?

How is an LLM trained (pretraining)?Tap to reveal

1. Take a huge amount of text and cut it into token sequences. 2. At every position, the model predicts the next token. 3. Measure the error with cross-entropy loss. 4. Backpropagation computes how each parameter should change; an optimizer (AdamW) nudges them. 5. Repeat for trillions of tokens.

It's 'self-supervised': the text is its own answer key, so no human labels are needed.

← PreviousNext card →