Apache Spark on Kubernetes - Lessons Learned from Launching Millions of Spark Executors

Overview

Explore lessons learned from launching millions of Spark executors on Kubernetes in this 35-minute conference talk by Databricks. Dive into Apple's approach to supporting enormous Spark workloads for cloud services, covering orchestration techniques across Mesos and Kubernetes, private and on-premise infrastructure. Learn about effective monitoring systems, resource requirement tuning, and execution analysis. Gain insights on optimizing Kubernetes for Spark workloads, implementing granular concurrency checks, mitigating cluster storage stress, and utilizing dynamic allocation. Discover strategies for push-button cloud management and scaling up Spark on Kubernetes to support varying workload patterns across multi-cloud environments.

Syllabus

Intro
Data Platform
Elastic Self Service Spark
Code to Deployment
Security
Monitoring
Orchestration Architecture
Varying Workload Pattern
One Interface over Multi-Cloud
Optimize Kubernetes for Spark Workload
Granular Concurrency Check at Orchestration
Avoid Partially Running Applications
Timeout Partially Running Applications
Mitigate Cluster Storage Stress
Utilization-based Allocation Recommendation
Dynamic Allocation
Push-button Cloud Management
Scale up Spark on Kubernetes