Trends in Deep Learning Hardware

Explore the evolution and future of deep learning hardware in this comprehensive lecture by NVIDIA's Chief Scientist, Bill Dally. Delve into the critical role of powerful, efficient computing hardware in enabling the current resurgence of artificial intelligence, including generative AI like ChatGPT. Trace the remarkable 1000x improvement in GPU efficiency for deep learning inference over the past decade, with a focus on advancements in data representation from FP32 to Int8 and FP8. Gain insights into upcoming innovations in number representation, including logarithmic representation, optimal clipping, and per-vector quantization. Learn about Dally's contributions to parallel computing architecture, signaling, routing, and synchronization technology, as well as his work on experimental systems like the J-Machine and M-Machine. Understand the historical context of deep learning algorithms and the pivotal role of GPUs in making this technology practical. This lecture, part of the Allen School Distinguished Lecture Series, offers a unique opportunity to learn from a leading expert in the field of computer architecture and deep learning hardware.