Practice
Learn
← Quit
0 / 0 correct
LLMs & AI
Hard
1 of 10
Why is single-request decoding on a GPU usually memory-bandwidth-bound?
A
GPUs have too little compute
B
Tokenization is slow
C
Each new token must read all model weights from GPU memory while doing relatively little math per byte
D
The KV cache is stored on disk