Benefits of Deep and Narrow Neural Networks for Approximation and Memorization

Explore a technical lecture that delves into two fundamental questions about neural networks' expressive power, focusing on deep and narrow architectures. Learn how depth proves more crucial than width in neural network approximation capabilities, with proof that bounded-width networks can approximate any function achievable by non-bounded width networks with only linear growth in neurons. Discover groundbreaking findings on neural network memorization capacity, demonstrating that O(√N) parameters suffice for memorizing N labeled points - an optimal result requiring significant network depth. Presented by Gilad Yehudai, a Ph.D. student at the Weizmann Institute of Science, whose expertise spans deep learning theory, optimization, and work with industry leaders like Google and NVIDIA.