Overview
Syllabus
Intro
What do we mean? "Memorizing training examples"
Empirical Example Memorization
Space, Information, and Deep Learning
Important preliminary: Shannon's mutual information
Theorem: Memorizing entire examples
Tasks: Mixtures of subpopulations
Tasks: Per-subpopulation distributions
Proof: Lower bounds via singletons
Experiments: Logistic regression and neural network
Setup: Learning from a stream of examples
Theorem: How Much Space?
Theorem: Example Memorization
Tasks: Space Lower Bounds for Natural Models
Proof: Structure and Overview
Proof: Requirements for distinguishing one bit
Main theorems and implications
Directions for future work
Memorize when you can't identify relevant information
Taught by
Google TechTalks