Building a 5-Exaflop Supercomputer for Meta-AI Research and Large-Scale Model Training

Explore the architecture, construction, and operation of Meta's cutting-edge AI Research SuperCluster in this 35-minute conference talk from SREcon23 Europe/Middle East/Africa. Discover how a small, geographically distributed team of Software and Production Engineers (SRE) collaborated to build and manage one of the world's largest AI supercomputers, boasting 16,000 GPUs and 5 exaflops of compute power. Gain insights into the challenges and solutions involved in supporting large-scale model training, including the recently released Llama series, with a focus on the InfiniBand interconnect, high-performance storage systems, and emerging monitoring and observability needs. Learn valuable lessons from Meta's experience in pushing the boundaries of AI research infrastructure and team collaboration.