← Tutorials3 / 17
Build a tiny LLM on your CPU
Step 1: how do we turn text into tokens?Tap to reveal
We use the simplest tokenizer: every distinct character is a token, numbered in sorted order. This text has 24 characters (letters, space, '.' and newline), so the vocabulary is 24.
Real LLMs use sub-word tokens (BPE), but the idea is the same: text in, integer IDs out, and back again.
struct Tokenizer { chars: Vec<char> }
fn encode(&self, text: &str) -> Vec<usize> {
text.chars()
.filter_map(|c| self.chars.binary_search(&c).ok())
.collect()
}