Cloud-Native Apache Spark Scheduling with YuniKorn on Kubernetes

Explore cloud-native Apache Spark scheduling using YuniKorn Scheduler in this 36-minute conference talk from Databricks. Dive into the architecture of cloud-native infrastructure and learn how YuniKorn, an open-source resource scheduler, redefines resource scheduling in the cloud. Discover how to manage quotas, resource sharing, and auto-scaling for efficient scheduling of large-scale Spark jobs on Kubernetes. Gain insights into Lyft and Cloudera's experiences with next-generation cloud-native infrastructure, and understand the challenges and solutions for running Spark on Kubernetes. Learn about YuniKorn's advantages over default schedulers, including job ordering, resource quota management, and fairness in queue allocation. Compare YuniKorn with other Kubernetes schedulers and explore its management console. Get an overview of YuniKorn's current status, community involvement, roadmap, and vision for resource management in big data and machine learning environments.