Explore the dynamics of gradient descent in deep learning through a comprehensive lecture that delves into optimization and generalization. Analyze the interplay between classical machine learning and deep learning approaches, focusing on gradient flow and end-to-end dynamics. Investigate implicit preconditioning, deep matrix factorization, and the dynamics of singular values in matrix completion problems. Gain insights into the role of nonlinearity in deep learning optimization and generalization, presented by Nadav Cohen from Tel-Aviv University as part of the "Learning and Testing in High Dimensions" series at the Simons Institute.
Analyzing Optimization and Generalization in Deep Learning via Dynamics of Gradient Descent
Overview
Syllabus
Introduction
What is Optimization Generalization
Classical Machine Learning
Deep Learning
Content
Gradient Flow
Endtoend Dynamics
Conventional Approach
Implicit Preconditioning
Gradient Descent
Depth
Matrix Completion
Deep Matrix Factorization
Experiments
Dynamics of Singular Values
Matrix Completion Problem
Singular Value Dynamics
Recap
Nonlinearity
Taught by
Simons Institute
Related Courses
-
Analyzing Optimization and Generalization in Deep Learning via Trajectories of Gradient Descent
-
Deep Learning - IIT Ropar
-
Deep Learning Interview Prep Course
-
Improving Deep Neural Networks: Hyperparameter Tuning, Regularization and Optimization
5.0 -
Learning Deep Matrix Factorizations Via Gradient Descent - Implicit Bias Towards Low Rank
-
On the Foundations of Deep Learning - SGD, Overparametrization, and Generalization
