3000+ Courses from California Community Colleges

Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Apache Spark on Kubernetes - Lessons Learned from Launching Millions of Spark Executors

Databricks via YouTube

Start learning Write review

Overview

Explore lessons learned from launching millions of Spark executors on Kubernetes in this 35-minute conference talk by Databricks. Dive into Apple's approach to supporting enormous Spark workloads for cloud services, covering orchestration techniques across Mesos and Kubernetes, private and on-premise infrastructure. Learn about effective monitoring systems, resource requirement tuning, and execution analysis. Gain insights on optimizing Kubernetes for Spark workloads, implementing granular concurrency checks, mitigating cluster storage stress, and utilizing dynamic allocation. Discover strategies for push-button cloud management and scaling up Spark on Kubernetes to support varying workload patterns across multi-cloud environments.

Syllabus

Intro
Data Platform
Elastic Self Service Spark
Code to Deployment
Security
Monitoring
Orchestration Architecture
Varying Workload Pattern
One Interface over Multi-Cloud
Optimize Kubernetes for Spark Workload
Granular Concurrency Check at Orchestration
Avoid Partially Running Applications
Timeout Partially Running Applications
Mitigate Cluster Storage Stress
Utilization-based Allocation Recommendation
Dynamic Allocation
Push-button Cloud Management
Scale up Spark on Kubernetes

Taught by

Databricks

Reviews

Start your review of Apache Spark on Kubernetes - Lessons Learned from Launching Millions of Spark Executors

Start learning

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Sign up for free
Someone learning on their laptop while sitting on the floor.