Explore a 14-minute conference talk from USENIX NSDI '23 that introduces ModelKeeper, an innovative system for accelerating deep neural network (DNN) training. Learn how this automated training warmup approach repurposes previously-trained models in shared clusters to jump-start new training jobs. Discover the key insights behind ModelKeeper, including its ability to identify architectural similarities between models, select suitable parent models, and perform structure-aware weight transformations. Understand how this system can significantly reduce training time across various computer vision and natural language processing models without compromising accuracy. Gain valuable insights into improving efficiency in machine learning workflows, particularly in environments with multiple neural architecture search and training jobs.
ModelKeeper - Accelerating DNN Training via Automated Training Warmup
Overview
Syllabus
NSDI '23 - ModelKeeper: Accelerating DNN Training via Automated Training Warmup
Taught by
USENIX
Related Courses
-
Zeus - Understanding and Optimizing GPU Energy Consumption of DNN Training
-
Bamboo - Making Preemptible Instances Resilient for Affordable Training of Large DNNs
-
TopoOpt - Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs
-
Accelerating Collective Communication in Data Parallel Training across Deep Learning Frameworks
-
Refurbish Your Training Data - Reusing Partially Augmented Samples for Faster Deep Neural Network Training
-
Unity - Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization
