Scaling Distributed XGBoost and Parallel Data Ingestion with Ray - FlightAware Case Study

Explore how FlightAware leverages Ray and AWS to scale distributed XGBoost training and parallel data ingestion for their runway prediction model. Dive into the process of building a cost-effective, scalable solution that efficiently processes terabytes of training data from S3 into distributed memory. Learn about organizing training data, configuring fault-tolerant and elastic Ray clusters, utilizing Amazon Lustre for FSx filesystem, and tracking metrics with MLFlow. Gain insights into optimizing costs and training time through practical tips and tricks discovered during the implementation. This 30-minute talk by Anyscale showcases the power of Ray in handling vast amounts of global aircraft data and demonstrates how to build an efficient distributed XGBoost training system for large-scale machine learning applications.