← Tutorials9 / 17

Build a tiny LLM on your CPU

How does gradient descent learn W?Tap to reveal

For softmax followed by cross-entropy, the gradient with respect to the logits is beautifully simple: predicted probabilities minus 1 at the true character (p − onehot).

Average that over all training pairs and move W a small step against it. Repeat 300 times.

let p = softmax(&self.logits(prev));
for (j, pj) in p.iter().enumerate() {
    grad[prev * v + j] += pj / n;
}
grad[prev * v + next] -= 1.0 / n;
// then: w -= lr * grad
← PreviousNext card →