← Tutorials16 / 17

Build a tiny LLM on your CPU

How do we get from this toy to a GPT?Tap to reveal

Each step is an idea you now know, scaled up: - 3 fixed characters of context become self-attention over thousands of tokens. - One hidden layer becomes dozens of transformer blocks. - Characters become BPE sub-word tokens. - 632 characters become trillions of tokens. - Plain SGD becomes AdamW, on many GPUs. - The trained model is then fine-tuned to chat (SFT, then preference tuning).

← PreviousNext card →