Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Building a Data Platform with Apache Spark on Kubernetes

WeAreDevelopers via YouTube

Overview

Explore the challenges and solutions of building a data platform using Apache Spark on Kubernetes in this 31-minute conference talk. Learn how PUBG Corporation migrated its on-demand data analytics platform to Spark on Kubernetes, serving millions of online gamers. Discover the Sphynx project, which manages on-demand Spark clusters and Jupyter Notebooks as containerized applications on Kubernetes. Gain insights into the main log pipeline, Apache Spark layer platform, batch systems, and data system domain. Understand Kubernetes deployment, scheduling, and platform architecture. Delve into workflows, best practices, monitoring strategies, and future work considerations. Walk away with key takeaways for implementing Spark on Kubernetes in large-scale data processing environments.

Syllabus

Introduction
Overview
Main Log Pipeline
Apache Spark
Layer Platform
Notebooks
Batch System
Spark Platform
Data System Domain
Problems
What is Kubernetes
Kubernetes Deployment
Kubernetes Scheduler
Platform Architecture
Workflow
Best Sauce
Challenges
Monitoring
Future Work
Key Takeaways
Questions

Taught by

WeAreDevelopers

Reviews

Start your review of Building a Data Platform with Apache Spark on Kubernetes

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.