Implicit and Explicit Regularization in Deep Neural Networks

Explore the theoretical underpinnings of deep learning in this 37-minute lecture by Babak Hassibi from the California Institute of Technology. Delve into the success of deep neural networks, focusing on the crucial role of stochastic descent methods in achieving good solutions that generalize well. Connect learning algorithms like stochastic gradient descent (SGD) and stochastic mirror descent (SMD) to H-infinity control, explaining their convergence and implicit regularization behavior in over-parameterized scenarios. Gain insights into the "blessing of dimensionality" phenomenon and learn about a new algorithm, regularized SMD (RSMD), which offers superior generalization performance for noisy datasets. Examine topics such as supervised learning, local optimization, prediction error, Bregman divergence, and the distribution of weights in neural networks.