← Tutorials13 / 17
Build a tiny LLM on your CPU
How does the MLP learn? (backpropagation)Tap to reveal
Same recipe, more layers. Start with (p − onehot) at the output, then apply the chain rule backwards: - through W2 into the hidden layer, - through tanh (its derivative is 1 − tanh²), - through W1 into the inputs, - and into the embedding rows that were used.
Each weight moves a little against its gradient after every example (stochastic gradient descent).
let dpre: Vec<f32> = (0..h)
.map(|j| dh[j] * (1.0 - f.h[j] * f.h[j]))
.collect();
// ... dx flows back into self.emb[t * d + i]