The platform may need to allocate hardware, start a process, download weights, compile kernels, and warm caches. Prewarming, minimum replicas, and smaller artifacts trade idle resources for faster first requests.
A cold start is added request latency caused by creating a serving instance or loading a model that is not already ready.
The platform may need to allocate hardware, start a process, download weights, compile kernels, and warm caches. Prewarming, minimum replicas, and smaller artifacts trade idle resources for faster first requests.