← Tutorials15 / 22
What is an LLM?
What is cross-entropy loss, intuitively?Tap to reveal
How surprised the model is by the true next token: loss = −ln(probability it gave the right token).
Guessing uniformly over 50,000 tokens gives ln(50,000) ≈ 10.8. Training drives this down; strong models reach around 2 on typical text. In the next tutorial you'll watch a tiny model's loss fall from 3.18 to 0.49.