SHEPHERD - Serving DNNs in the Wild

Explore a groundbreaking model serving system called SHEPHERD in this 15-minute conference talk from NSDI '23. Discover how SHEPHERD tackles the challenges of scalability, high system goodput, and maximum resource utilization across compute units for inference requests in interactive web services. Learn about its innovative two-level design that separates planning and serving modules, leveraging request stream aggregation for improved predictability and resource utilization. Understand the novel online algorithm employed by SHEPHERD for guaranteed goodput under unpredictable workloads, utilizing preemptions and model-specific batching properties. Gain insights into the system's performance, which achieves up to 18.1X higher goodput and 1.8X better utilization compared to prior state-of-the-art solutions, while scaling to hundreds of workers.