Completed
Intro - focusing on the 3D parallelism!
Class Central Classrooms beta
YouTube videos curated by Class Central.
Classroom Contents
BigScience BLOOM - 3D Parallelism Explained - Large Language Models - ML Coding Series
Automatically move to the next video in the Classroom when playback concludes
- 1 Intro - focusing on the 3D parallelism!
- 2 Quick setup
- 3 Stepping through the eval script
- 4 3D paralellism - model construction
- 5 Sharding the embedding table model parallelism
- 6 Sharding the transformer layer
- 7 LayerNorm fused kernels
- 8 Sharding the attention layer
- 9 ColumnParallel and RowParallel sharding
- 10 Synchronizing input and output embedding tables
- 11 Building the dataset data parallelism
- 12 3D parallelism - forward pass
- 13 Pipeline parallelism communication
- 14 Pass through the sharded embedding table
- 15 Pass through the sharded transformer layer
- 16 Sharded logit and cross-entropy computation
- 17 Recap
- 18 Outro
