Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Generalization in Deep Learning Through the Lens of Implicit Rank Minimization

Hausdorff Center for Mathematics via YouTube

Overview

Explore a lecture on the implicit regularization in deep learning through the lens of rank minimization. Delve into the theoretical analysis of matrix and tensor factorizations, equivalent to certain linear and non-linear neural networks. Discover how gradient-based optimization tends to fit training data with predictors of low complexity, leading to generalization. Examine dynamical characterizations establishing implicit regularization towards low matrix and tensor ranks, challenging prior beliefs about norm minimization. Consider the implications of these findings for both theory and practice in modern deep learning, highlighting the potential of ranks to explain and improve generalization. Learn about matrix completion, tensor completion, and their connections to linear and non-linear neural networks. Investigate practical applications, including rank minimization in neural network layers and potential explanations for generalization on natural data.

Syllabus

Intro
Generalization via Bis-Variance Tradeoff
Generalization in Deep Learning
Linear Models: Implicit Norm Minimization Linear Regression
Implicit Norm Minimization In Deep Learning?
Perspective: Implicit Rank Minimization
Outline
Matrix Completion Two-Dimensional Prediction
MF Linear NN
Conjecture: Implicit Nuclear Norm Minimization
Dynamical Analysis of Implicit Regularization in MF
Implicit Regularization in MF Norm Minimization Does the implicit regularization in MF minimize a norm?
Drawbacks of Studying MF
Tensor Completion Multi-Dimensional Prediction
TF Shallow Non-Linear Convolutional NN
Dynamical Analysis of Implicit Regularization in TF
Analogy Between Implicit Regularizations
HTF Deep Non-Linear CNN TF does not account for depth
Dynamical Analysis of Implicit Regularization in HTF
Practical Application: Rank Minimization in NN Layers
Potential Explanation for Generalization on Natural Data
Countering Locality of CNNs via Regularization
Recap
Implicit Rank Minimization in Deep Learning

Taught by

Hausdorff Center for Mathematics

Reviews

Start your review of Generalization in Deep Learning Through the Lens of Implicit Rank Minimization

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.