Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Managing Multi-Cloud Apache Spark on Kubernetes

CNCF [Cloud Native Computing Foundation] via YouTube

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Explore the challenges and solutions of managing multi-cloud Apache Spark on Kubernetes in this 31-minute conference talk by Ilan Filonenko and Aki Sukegawa from Bloomberg. Dive into Bloomberg's journey of building multi-cloud quant platforms on Kubernetes for financial applications with integrated data science capabilities. Learn about the complexities of managing data science infrastructure across multiple cloud environments, focusing on Apache Spark. Discover strategies for effective Spark infrastructure management spanning bare-metal and public cloud platforms. Examine approaches to auto-scaling, scheduling, preemption, and security in Kubernetes. Gain insights into observability techniques, including methods to surface cluster information to diverse Spark end-users using native Kubernetes resources such as node autoscalers, controllers, and custom PodConditions. Follow the speakers as they discuss user stories, complications, and solutions, exploring topics like custom resources, cluster scaling, event handling, and PodStatus Controller behavior.

Syllabus

Intro
Background (Kubernetes)
Background (Apache Spark)
Background (Spark)
User Stories (Complications)
User Stories (Solutions)
Why custom resource (CR)
Storing information, where to?
First: Cluster scaling up
Cluster autoscaler events
Controller to look up event objects
Next: Scaling down, OOM, etc.
Keeping pods?
Kubernetes custom resource (CR)
PodStatus Controller behavior
Extra: Declarative copying
Extensions

Taught by

CNCF [Cloud Native Computing Foundation]

Reviews

Start your review of Managing Multi-Cloud Apache Spark on Kubernetes

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.