← Tutorials4 / 17

Build a tiny LLM on your CPU

How will we know if a model is any good?Tap to reveal

With the same measure real LLMs use: average cross-entropy loss, −ln(probability the model gave the true next character). Lower is better.

The baseline: guessing uniformly over 24 characters gives ln(24) = 3.178. Every model we build must beat that.

fn mean_loss(items: impl Iterator<Item = (Vec<f32>, usize)>) -> f32 {
    // for each (logits, true_next): add -ln(softmax(logits)[true_next])
}
← PreviousNext card →