← Quit0 / 0 correct
What is the KV cache in LLM inference?
step t: new token ─▶ q_t
cached: K[0..t-1], V[0..t-1]
append k_t, v_t to the cache
attend q_t over K[0..t], V[0..t]What is the KV cache in LLM inference?
step t: new token ─▶ q_t
cached: K[0..t-1], V[0..t-1]
append k_t, v_t to the cache
attend q_t over K[0..t], V[0..t]