The autoregressive phase of inference where the model generates one token at a time, each step attending to all previous tokens via the KV cache — its speed determines tokens-per-second.
The autoregressive phase where the model generates one token at a time using the KV cache.
The autoregressive phase of inference where the model generates one token at a time, each step attending to all previous tokens via the KV cache — its speed determines tokens-per-second.