← Tutorials5 / 17
Build a tiny LLM on your CPU
Step 2: what's the simplest possible language model?Tap to reveal
A bigram model: predict the next character from only the current one, by counting.
counts[a][b] = how often b followed a in the text. Divide a row by its total and you get the probabilities for what comes after a. We start every count at 1 ('add-one smoothing') so no character is ever impossible.
let mut counts = vec![1.0; vocab * vocab];
for pair in ids.windows(2) {
counts[pair[0] * vocab + pair[1]] += 1.0;
}