Toward Theoretical Understanding of Deep Learning - Lecture 2
International Centre for Theoretical Sciences via YouTube
Overview
Syllabus
Date & Time: Tuesday, 12 February,
Date & Time: Tuesday, 12 February,
Date & Time: Wednesday, 13 February,
Start
Toward theoretical understanding of deep learning
Machine learning ML: A new kind of science
Recap:
Training via Gradient Descent "natural algorithm"
Subcase: deep learning*
Brief history: networks of "artificial neurons"
Some questions
Part 1: Why overparameterization and/or overprovisioning?
Overprovisioning may help optimization part 1: a folklore experiment
Overprovisioning can help part 2: Allowing more
Acceleration effect of increasing depth
But textbooks warn us: Larger models can "Overfit"
Popular belief/conjecture
Noise stability: understanding one layer no nonlinearity
Proof sketch : Noise stability -deep net can be made low-dimensional
The Quantitative Bound
Correlation with Generalization qualitative check
Concluding thoughts on generalization
Part 2: Optimization in deep learning
Basic concepts
Curse of dimensionality
Gradient descent in unknown landscape.
Gradient descent in unknown landscape contd.
Evading saddle points..
Active area: Landscape Analysis
New trend: Trajectory Analysis
Trajectory Analysis contd
Unsupervised learning motivation: "Manifold assumption"
Unsupervised learning Motivation: "Manifold assumption" contd
Deep generative models
Generative Adversarial Nets GANs [Goodfellow et al. 2014]
What spoils a GANs trainer's day: Mode Collapse
Empirically detecting mode collapse Birthday Paradox Test
Estimated support size from well-known GANs
To wrap up....What to work on suggestions for theorists
Concluding thoughts
Advertisements
Q&A
Taught by
International Centre for Theoretical Sciences