Completed
Model Parallelism: An alternative to data parallelism
Class Central Classrooms beta
YouTube playlists curated by Class Central.
Classroom Contents
Generalized Pipeline Parallelism for DNN Training - PipeDream System Overview
Automatically move to the next video in the Classroom when playback concludes
- 1 Intro
- 2 Model Parallelism: An alternative to data parallelism
- 3 Pipelining in DNN training != Traditional pipelining
- 4 Challenge 1: Pipelining leads to weight version mismatches
- 5 Weight stashing: A solution to version mismatches
- 6 Challenge 2: How do we assign operators to pipeline stages?
- 7 Pipe Dream vs. Data Parallelism on Time-to-Accuracy
- 8 but modern Deep Neural Networks are becoming extremely large!
- 9 Double-buffered weight updates: weight semantics
- 10 2BW has weight update semantics similar to data parallelism
