Each new token attends to cached states and adds its own key and value. The cache reduces repeated computation but consumes memory that grows with sequence length, layers, and active requests.
A KV cache stores attention keys and values from earlier tokens so autoregressive generation does not recompute them at every step.
Each new token attends to cached states and adds its own key and value. The cache reduces repeated computation but consumes memory that grows with sequence length, layers, and active requests.