← Tutorials3 / 22
What is an LLM?
What is a token?Tap to reveal
"unbelievably fast" ┌──┬──────┬────┬──┬─────┐ │un│belie │vab │ly│ fast│ └──┴──────┴────┴──┴─────┘ each piece ─▶ one integer id (illustrative; splits vary by tokenizer)
A piece of text the model reads and writes as one unit: a word, part of a word, a symbol or a byte. Each token is an integer ID.
In English a token is about 4 characters (≈ ¾ of a word). Prices, context limits and speed are all measured in tokens.