Explore an innovative approach to automating model-parallel training for large deep learning models in this 18-minute conference talk from OSDI '22. Discover how Alpa generates execution plans that unify data, operator, and pipeline parallelism, addressing the limitations of existing model-parallel training systems. Learn about Alpa's hierarchical view of parallelisms, its new space for massive model-parallel execution plans, and the compilation passes designed to derive efficient parallel execution plans. Understand how Alpa's runtime orchestrates two-level parallel execution on distributed compute devices, and examine its performance compared to hand-tuned systems. Gain insights into Alpa's versatility in handling models with heterogeneous architectures and those without manually-designed plans. Access the source code and explore the potential of this groundbreaking approach to scaling out complex deep learning models on distributed systems.
Alpa - Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
Overview
Syllabus
OSDI '22 - Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
Taught by
USENIX
Related Courses
-
Alpa: Simple Large Model Training and Inference on Ray
-
AlpaServe - Statistical Multiplexing with Model Parallelism for Deep Learning Serving
-
Distributed Deep Graph Learning at Scale
-
Rammer - Enabling Holistic Deep Learning Compiler Optimizations with rTasks
-
nnScaler: Constraint-Guided Parallelization Plan Generation for Deep Learning Training
-
AdaEmbed - Adaptive Embedding for Large-Scale Recommendation Models
