Watch a technical conference presentation from HOTI (Hot Interconnects Symposium) exploring an innovative network architecture called Rail-only, designed specifically for training large language models with trillion parameters. Learn about this cost-effective yet high-performance networking solution presented by researchers Weiyang Wang, Manya Ghobadi, Kayvon Shakeri, Ying Zhang and Naader Hasani as part of the Technical Paper Session on Networks for Large Language Models. Discover how this approach addresses the networking challenges in training massive AI models while maintaining efficiency and performance in this 32-minute talk chaired by AMD's Shelby Lockhart.
Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters
HOTI - Hot Interconnects Symposium via YouTube
Overview
Syllabus
Day 1 09:00: Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Params
Taught by
HOTI - Hot Interconnects Symposium
Related Courses
-
Characterizing Communication in Distributed Parameter-Efficient Fine-Tuning for LLMs
-
Pretraining LLMs
-
Demystifying the Communication Characteristics for Distributed Transformer Models
-
Principles and Practice of Scalable and Distributed Deep Neural Networks Training and Inference
-
High Performance Machine Learning, Deep Learning, and Data Science - Principles and Practice
-
Powering Llama 3 - A Peek into Meta's Massive Infrastructure for Generative AI
