Swing: Short-cutting Rings for Higher Bandwidth Allreduce

Explore a groundbreaking algorithm for improving allreduce performance on torus networks in this 26-minute conference talk from NSDI 2024. Delve into the innovative Swing algorithm, presented by Daniele De Sensi from the Scalable Parallel Computing Lab at ETH Zurich. Learn how Swing enhances allreduce bandwidth by maintaining low distances between communicating nodes through swinging between torus directions. Discover the algorithm's impressive performance gains, outperforming existing methods by up to 3x for various vector sizes and torus topologies. Gain insights into the potential impact of this research on machine learning workloads and high-performance computing systems, including Google TPUs, Amazon Trainium devices, and Top500 supercomputers.