← Tutorials4 / 22

What is an LLM?

Why do LLMs use sub-word tokens (BPE) instead of words or characters?Tap to reveal

Characters make sequences very long (slow, and harder to learn from). Whole words need a huge vocabulary and can't handle new words or typos.

Byte-Pair Encoding starts from bytes and repeatedly merges the most frequent adjacent pair into a new token, giving a vocabulary of ~50k–200k pieces: common words become one token, rare words split into a few, and any text can be encoded.

← PreviousNext card →