← Tutorials5 / 17

Build a tiny LLM on your CPU

Step 2: what's the simplest possible language model?Tap to reveal

A bigram model: predict the next character from only the current one, by counting.

counts[a][b] = how often b followed a in the text. Divide a row by its total and you get the probabilities for what comes after a. We start every count at 1 ('add-one smoothing') so no character is ever impossible.

let mut counts = vec![1.0; vocab * vocab];
for pair in ids.windows(2) {
    counts[pair[0] * vocab + pair[1]] += 1.0;
}
← PreviousNext card →