Explore recent developments in over-parametrized neural networks in this comprehensive lecture from the Deep Learning Boot Camp. Delve into topics such as non-convex optimization, overparametrization, and theoretical problems in optimal optimization. Examine the taxonomy of results, including skip connections, expressivity, and geometric results. Investigate one-point convexity, Taylor's theorem, and the Royer algorithm. Learn about second-order stationary points, optimality conditions, and global optimality in neural networks. Discover strategies for second-order descent and stationary points in learning networks. Gain valuable insights from Jason Lee of the University of Southern California as he presents cutting-edge research and concepts in the field of deep learning and neural networks.
Recent Developments in Over-parametrized Neural Networks, Part I
Overview
Syllabus
Introduction
Outline
Goals
Whats Hard
Whats Easy
Non convex optimization
Notation
Overparametrization
Experiments
Rule of Thumb
Theoretical Problem
Optimal Optimization
Taxonomy of Results
Skip Connections
Expressivity
Geometric Results
Geometry Results
Onepoint convexity
Taylors theorem
Royer algorithm
Randomness
Theorem
Secondorder Stationary Points
Neural Networks
Optimality Conditions
Global Optimal
Other Losses
NonDegenerate Critical Points
Three Strategies
Second Order Descent
Stationary Points
Learning Networks
Taught by
Simons Institute
Related Courses
-
Improving Deep Neural Networks: Hyperparameter Tuning, Regularization and Optimization
5.0 -
Recent Developments in Over-parametrized Neural Networks - Part II
-
Can Learning Theory Resist Deep Learning? - Francis Bach, INRIA
-
Gradient Descent on Infinitely Wide Neural Networks - Global Convergence and
-
Learning One-Hidden-Layer Neural Networks with Landscape Design
-
Optimization Landscape and Two-Layer Neural Networks - Rong Ge
