Explore fundamental concepts in statistical learning theory and their application to deep neural networks in this comprehensive tutorial. Delve into uniform laws of large numbers and their relationship to function class complexity. Focus on Rademacher complexity as a key measure, examining upper bounds for deep ReLU networks. Investigate the apparent contradictions between modern neural network behaviors and classical intuitions. Gain insights into neural network training from an optimization perspective, reviewing gradient descent analysis for convex and smooth objectives. Understand the Polyak-Lojasiewicz (PL) inequality and its relevance to neural network training. Examine the neural tangent kernel (NTK) regime and its approximation of neural network training. Learn two approaches to establishing PL inequalities for neural networks: a general method based on NTK approximation and a specific technique for linearly-separable data.
Statistical Learning Theory and Neural Networks
Overview
Syllabus
Tutorial: Statistical Learning Theory and Neural Networks I
Taught by
Simons Institute
Related Courses
-
Statistical Learning Theory and Neural Networks II
-
How Neural Networks Learn: Exploring Architecture, Gradient Descent, and Backpropagation
-
Learning and Generalization in Over-parametrized Neural Networks - Going Beyond Kernels
-
The Information Bottleneck Theory of Deep Neural Networks
-
The Interpolation Phase Transition in Neural Networks - Memorization and Generalization Lazy Training
-
A Function Space View of Overparameterized Neural Networks - Rebecca Willet, University of Chicago
