← Tutorials13 / 17

Build a tiny LLM on your CPU

How does the MLP learn? (backpropagation)Tap to reveal

Same recipe, more layers. Start with (p − onehot) at the output, then apply the chain rule backwards: - through W2 into the hidden layer, - through tanh (its derivative is 1 − tanh²), - through W1 into the inputs, - and into the embedding rows that were used.

Each weight moves a little against its gradient after every example (stochastic gradient descent).

let dpre: Vec<f32> = (0..h)
    .map(|j| dh[j] * (1.0 - f.h[j] * f.h[j]))
    .collect();
// ... dx flows back into self.emb[t * d + i]
← PreviousNext card →