â Tutorials4 / 17
Build a tiny LLM on your CPU
How will we know if a model is any good?Tap to reveal
With the same measure real LLMs use: average cross-entropy loss, âln(probability the model gave the true next character). Lower is better.
The baseline: guessing uniformly over 24 characters gives ln(24) = 3.178. Every model we build must beat that.
fn mean_loss(items: impl Iterator<Item = (Vec<f32>, usize)>) -> f32 {
// for each (logits, true_next): add -ln(softmax(logits)[true_next])
}