Klaviyo's Journey to Robust Model Serving with Ray Serve

Explore Klaviyo's journey in building a robust model serving platform using Ray Serve, as presented at Ray Summit 2024. Dive into the ML platform team's experience in creating DART (DAtascience RunTime), their model serving solution. Discover the rationale behind choosing Ray Serve, emphasizing its balance of flexibility, simplicity, and high availability. Gain insights into DART's architecture and how it empowers data scientists to deploy models efficiently. Examine key aspects of Ray Serve's internal architecture, including request routing and fault tolerance with KubeRay. Learn about DART's multi-cluster, multi-AZ setup for enhanced resilience. Benefit from practical advice on cluster sizing, traffic isolation, and avoiding common pitfalls when implementing Ray Serve. Gain valuable insights from Klaviyo's experience to develop serving platforms tailored to specific requirements. This 31-minute presentation offers a comprehensive look at leveraging Ray Serve for robust model serving in real-world applications.