Learn about the communication patterns and characteristics of distributed transformer models in this 34-minute technical conference presentation from the HOTI Hot Interconnects Symposium. Explore detailed insights from researchers Quentin Anthony, Benjamin Michalowicz, Jacob Hatef, Lang Xu, Mustafa Abduljabbar, Aamir Shafi, Hari Subramoni and Dhabaleswar Panda as they break down the complex communication requirements for running large-scale transformer architectures across distributed systems. Part of Technical Paper Session D focused on optimizing collective operations, this talk chaired by Craig Stunkel of Nvidia provides valuable understanding of the networking and communication challenges involved in training and deploying distributed transformer models.
Demystifying the Communication Characteristics for Distributed Transformer Models
HOTI - Hot Interconnects Symposium via YouTube
Overview
Syllabus
Day 2 13:00: Demystifying the Communication Characteristics for Distributed Transformer Models
Taught by
HOTI - Hot Interconnects Symposium
Related Courses
-
Characterizing Communication in Distributed Parameter-Efficient Fine-Tuning for LLMs
-
OHIO: Improving RDMA Network Scalability in MPI_Alltoall Through Optimized Communication Design
-
High Performance Machine Learning, Deep Learning, and Data Science - Principles and Practice
-
Principles and Practice of Scalable and Distributed Deep Neural Networks Training and Inference
-
Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters
-
Unified Collective Communication - A Library for CPU, GPU, and DPU Integration
