← Tutorials11 / 17

Build a tiny LLM on your CPU

Step 4: how do we give the model more context?Tap to reveal

We build a small multi-layer perceptron (MLP) that reads the last 3 characters:

1. Look up a 10-number embedding for each character. 2. Concatenate them into 30 numbers. 3. Pass them through a hidden layer of 64 tanh units. 4. Output 24 logits.

This is Bengio's 2003 neural language model: the direct ancestor of modern LLMs.

← PreviousNext card →