The stored key and value tensors from previous tokens that let the model attend to its full context without recomputing attention from scratch at each step — the main memory bottleneck for long contexts.
Stored key-value tensors from previous tokens that avoid recomputing attention at each step.
The stored key and value tensors from previous tokens that let the model attend to its full context without recomputing attention from scratch at each step — the main memory bottleneck for long contexts.