← Tutorials10 / 17
Build a tiny LLM on your CPU
What happened when we trained it?Tap to reveal
Loss falls from 3.178 (uniform) to 1.423: you're watching learning happen.
It ends slightly below the counting model (1.757) because counting added +1 to every pair, which blurs the statistics on such a small text. Both are the same kind of model, but gradient descent also works for models far too complex to 'count', which is the whole point.
2) neural bigram step 0 loss 3.178
2) neural bigram step 100 loss 1.547
2) neural bigram step 200 loss 1.457
2) neural bigram step 300 loss 1.423