The fundamental trade-off in inference serving: batching more requests increases total tokens processed per second (throughput) but can increase wait time for individual requests (latency).
The fundamental trade-off where batching increases total throughput but can increase per-request latency.
The fundamental trade-off in inference serving: batching more requests increases total tokens processed per second (throughput) but can increase wait time for individual requests (latency).