Explore the challenges and solutions of running Apache Spark on Kubernetes at scale in this insightful conference talk. Discover how to leverage open-source technologies like Apache YuniKorn, Spark K8s operator, and Cloud primitives to evolve ML data infrastructure in the cloud. Learn about considerations for multi-tenancy, observability, scalability, and cost-effectiveness when implementing Spark on Kubernetes for large-scale production environments. Gain valuable insights on overcoming roadblocks such as dev-ops complexity, multi-cluster management, job scheduling, and autoscaling to maximize data processing capabilities on the cloud.
Overview
Syllabus
Beyond Experimental: Spark on Kubernetes - Weiwei Yang, Apple
Taught by
CNCF [Cloud Native Computing Foundation]
Related Courses
-
Azure Kubernetes Service (AKS) application and cluster scalability
-
Cloud-Native Apache Spark Scheduling with YuniKorn on Kubernetes
-
Spark on Kubernetes: Building an Elastic Service - Apple's Approach
-
Managing Multi-Cloud Apache Spark on Kubernetes
-
Migrating Airflow-Based Apache Spark Jobs to Kubernetes - The Native Way
-
Elevating Kubeflow Spark Operator's Future: Best Practices and Enhancements
