Colocating Hadoop YARN with Kubernetes to Save Costs on Big Data

Explore how to colocate Hadoop YARN with Kubernetes to significantly reduce costs on big data operations in this 43-minute conference talk by Irvin Lim and Hailin Xiang from Shopee. Learn about the challenges of low resource utilization in Kubernetes clusters and the complexities of running online services alongside offline jobs. Discover innovative solutions for customizing and extending Linux Kernel, Container Runtime, Kubernetes Scheduler, and Kubelet to improve resource utilization while maintaining the performance of online services. Gain insights into overcoming limitations of default cgroup CFS and memory limits in real-world scenarios, and understand how to navigate Kubernetes restrictions on offline job scheduling to optimize computing resource costs for big data operations.