← Tutorials11 / 17
Build a tiny LLM on your CPU
Step 4: how do we give the model more context?Tap to reveal
'h' 'e' ' ' last 3 chars
│ │ │
emb emb emb 10 numbers each
└─────┼─────┘
concat (30)
│
hidden: tanh(x·W1+b1) 64 units
│
logits: h·W2 + b2 24 scores
│
softmax ─▶ next charWe build a small multi-layer perceptron (MLP) that reads the last 3 characters:
1. Look up a 10-number embedding for each character. 2. Concatenate them into 30 numbers. 3. Pass them through a hidden layer of 64 tanh units. 4. Output 24 logits.
This is Bengio's 2003 neural language model: the direct ancestor of modern LLMs.