← Tutorials10 / 17

Build a tiny LLM on your CPU

What happened when we trained it?Tap to reveal

Loss falls from 3.178 (uniform) to 1.423: you're watching learning happen.

It ends slightly below the counting model (1.757) because counting added +1 to every pair, which blurs the statistics on such a small text. Both are the same kind of model, but gradient descent also works for models far too complex to 'count', which is the whole point.

2) neural bigram   step   0  loss 3.178
2) neural bigram   step 100  loss 1.547
2) neural bigram   step 200  loss 1.457
2) neural bigram   step 300  loss 1.423
← PreviousNext card →