← Tutorials8 / 17
Build a tiny LLM on your CPU
Step 3: how do we turn the bigram model into a neural network?Tap to reveal
Replace the counts with a table of learnable numbers W (24 × 24 logits), starting at zero, which means a uniform guess.
The prediction after character a is softmax(W[a]). Instead of counting, we find good values for W with gradient descent, the same method that trains GPT.
struct NeuralBigram { vocab: usize, w: Vec<f32> }
fn logits(&self, prev: usize) -> Vec<f32> {
self.w[prev * self.vocab..(prev + 1) * self.vocab].to_vec()
}