It manages request queues, batching, memory, decoding, and communication with accelerators. Standard APIs and model repositories let applications use different versions while the server handles execution details.
An inference server is software that loads models and schedules prediction work on compute hardware.
It manages request queues, batching, memory, decoding, and communication with accelerators. Standard APIs and model repositories let applications use different versions while the server handles execution details.