← Tutorials8 / 17

Build a tiny LLM on your CPU

Step 3: how do we turn the bigram model into a neural network?Tap to reveal

Replace the counts with a table of learnable numbers W (24 × 24 logits), starting at zero, which means a uniform guess.

The prediction after character a is softmax(W[a]). Instead of counting, we find good values for W with gradient descent, the same method that trains GPT.

struct NeuralBigram { vocab: usize, w: Vec<f32> }

fn logits(&self, prev: usize) -> Vec<f32> {
    self.w[prev * self.vocab..(prev + 1) * self.vocab].to_vec()
}
← PreviousNext card →