Explore cutting-edge techniques for accelerating the training of large language models in this 19-minute conference talk from USENIX ATC '24. Dive into novel strategies for efficient activation rematerialization and optimal hybrid parallelism presented by researchers from Kuaishou Technology. Learn about Pipeline-Parallel-Aware Offloading, which maximizes host memory utilization for storing activations, and Compute-Memory Balanced Checkpointing, which balances activation memory and computational efficiency. Discover an efficient searching method for optimizing hybrid parallelism parameters, considering both offloading and checkpointing for optimal performance. Examine the results of extensive experiments on public benchmarks, showcasing significant improvements in Model FLOPs Utilization (MFU) for large-scale models with varying context window sizes. Gain insights into the latest advancements in training large-scale models, focusing on optimizing activation strategies and exploring various parallel training options to enhance the balance between computation and memory utilization.
Accelerating the Training of Large Language Models - Efficient Activation Rematerialization and Optimal Hybrid Parallelism
Overview
Syllabus
USENIX ATC '24 - Accelerating the Training of Large Language Models using Efficient Activation...
Taught by
USENIX
Related Courses
-
Ultimate Guide to Scaling ML Models - Megatron-LM - ZeRO - DeepSpeed - Mixed Precision
-
Efficient Large-Scale Language Model Training on GPU Clusters
-
FwdLLM: Efficient Federated Finetuning of Large Language Models with Perturbed Inferences
-
Scaling ML Workloads with PyTorch
-
MegaScale - Scaling Large Language Model Training to More Than 10,000 GPUs
-
PUZZLE: Efficiently Aligning Large Language Models through Light-Weight Context Switching
