Dive into the architecture and deployment of Zeus, Uber's highly scalable and distributed shuffle as a service powering all data processing at the company. Explore how this ground-up solution supports hundreds of thousands of jobs and millions of containers, shuffling petabytes of data. Learn about Zeus' paradigm-shifting approach to external shuffle, resulting in improved performance for most jobs despite remote data writing. Compare Zeus' performance with different storage systems backed by external shuffle, such as NFS and HDFS. Discover the integration of Zeus with Spark and how it contrasts with Spark's built-in sort-based shuffle mechanism. Gain insights into the future roadmap and plans for Zeus, and understand its impact on addressing hardware failures, reliability, and scalability challenges in one of the largest Spark and Hive clusters in the industry.
Overview
Syllabus
Introduction
External Shuffle Service
Deep Dive
Design Principle
Horizontal Scalable
Network Connections
Network Latency
Compression
Network IO
Connection
Asynchronous Commit
Tolerance
Local States
Spark Compatibility
Metrics
Production Quality
Data Management
Summary
Taught by
Databricks
