â Tutorials4 / 22
What is an LLM?
Why do LLMs use sub-word tokens (BPE) instead of words or characters?Tap to reveal
Characters make sequences very long (slow, and harder to learn from). Whole words need a huge vocabulary and can't handle new words or typos.
Byte-Pair Encoding starts from bytes and repeatedly merges the most frequent adjacent pair into a new token, giving a vocabulary of ~50kâ200k pieces: common words become one token, rare words split into a few, and any text can be encoded.