← Tutorials16 / 17
Build a tiny LLM on your CPU
How do we get from this toy to a GPT?Tap to reveal
counting bigram (step 2) ▼ gradient descent neural bigram (step 3) ▼ embeddings + context MLP language model (step 4) ▼ attention, depth, data transformer (GPT) ▼ SFT + RLHF/DPO chat assistant
Each step is an idea you now know, scaled up: - 3 fixed characters of context become self-attention over thousands of tokens. - One hidden layer becomes dozens of transformer blocks. - Characters become BPE sub-word tokens. - 632 characters become trillions of tokens. - Plain SGD becomes AdamW, on many GPUs. - The trained model is then fine-tuned to chat (SFT, then preference tuning).