Generalized Pipeline Parallelism for DNN Training - PipeDream System Overview

Generalized Pipeline Parallelism for DNN Training - PipeDream System Overview

Databricks via YouTube Direct link

Intro

1 of 10

1 of 10

Intro

Class Central Classrooms beta

YouTube videos curated by Class Central.

Classroom Contents

Generalized Pipeline Parallelism for DNN Training - PipeDream System Overview

Automatically move to the next video in the Classroom when playback concludes

  1. 1 Intro
  2. 2 Model Parallelism: An alternative to data parallelism
  3. 3 Pipelining in DNN training != Traditional pipelining
  4. 4 Challenge 1: Pipelining leads to weight version mismatches
  5. 5 Weight stashing: A solution to version mismatches
  6. 6 Challenge 2: How do we assign operators to pipeline stages?
  7. 7 Pipe Dream vs. Data Parallelism on Time-to-Accuracy
  8. 8 but modern Deep Neural Networks are becoming extremely large!
  9. 9 Double-buffered weight updates: weight semantics
  10. 10 2BW has weight update semantics similar to data parallelism

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.