The latency from sending a request to receiving the first output token — dominated by prefill time and queuing delay.
The latency from sending a request to receiving the first output token.
The latency from sending a request to receiving the first output token — dominated by prefill time and queuing delay.