The scheduler groups compatible decoding steps rather than waiting for an entire fixed batch to complete. This keeps accelerators busier under varied sequence lengths but requires careful cache and fairness management.
Continuous batching dynamically adds and removes generation requests from an active inference batch as sequences start and finish.
The scheduler groups compatible decoding steps rather than waiting for an entire fixed batch to complete. This keeps accelerators busier under varied sequence lengths but requires careful cache and fairness management.