← Tutorials15 / 22

What is an LLM?

What is cross-entropy loss, intuitively?Tap to reveal

How surprised the model is by the true next token: loss = −ln(probability it gave the right token).

Guessing uniformly over 50,000 tokens gives ln(50,000) ≈ 10.8. Training drives this down; strong models reach around 2 on typical text. In the next tutorial you'll watch a tiny model's loss fall from 3.18 to 0.49.

← PreviousNext card →