← Tutorials3 / 17

Build a tiny LLM on your CPU

Step 1: how do we turn text into tokens?Tap to reveal

We use the simplest tokenizer: every distinct character is a token, numbered in sorted order. This text has 24 characters (letters, space, '.' and newline), so the vocabulary is 24.

Real LLMs use sub-word tokens (BPE), but the idea is the same: text in, integer IDs out, and back again.

struct Tokenizer { chars: Vec<char> }

fn encode(&self, text: &str) -> Vec<usize> {
    text.chars()
        .filter_map(|c| self.chars.binary_search(&c).ok())
        .collect()
}
← PreviousNext card →