← Tutorials9 / 17
Build a tiny LLM on your CPU
How does gradient descent learn W?Tap to reveal
For softmax followed by cross-entropy, the gradient with respect to the logits is beautifully simple: predicted probabilities minus 1 at the true character (p − onehot).
Average that over all training pairs and move W a small step against it. Repeat 300 times.
let p = softmax(&self.logits(prev));
for (j, pj) in p.iter().enumerate() {
grad[prev * v + j] += pj / n;
}
grad[prev * v + next] -= 1.0 / n;
// then: w -= lr * grad