SGD in the Large - Average-Case Analysis, Asymptotics, and Stepsize Criticality

Explore the intricacies of Stochastic Gradient Descent (SGD) in a comprehensive lecture delivered by Courtney Paquette from McGill University at the Machine Learning Advances and Applications Seminar. Delve into average-case analysis, asymptotics, and stepsize criticality as key components of SGD. Examine optimization problems, average-case complexity, and the role of randomness and distribution in SGD. Investigate the SGD congruence theorem and its implications for worst-case scenarios. Uncover the nuances of stepsize criticality and its impact on average-case complexity. Learn about stochastic momentum, the stochastic heavy ball method, and the significance of momentum parameters. Discover dimension-dependent momentum and its applications in logistic regression. Gain valuable insights into the average-case analysis of SGD and its relevance in machine learning applications.