Infinite Limits and Scaling Laws of Neural Networks

Explore the cutting-edge research on neural network scaling laws and infinite parameter limits in this one-hour lecture by Blake Bordelon from Harvard University. Delivered at IPAM's Theory and Practice of Deep Learning Workshop, the talk delves into the breakthroughs in computer vision and natural language processing enabled by scaling up deep learning models. Examine infinite parameter limits of deep neural networks that preserve representation learning and understand the convergence rate of finite models to these limits. Discover how dynamical mean field theory methods provide an asymptotic description of learning dynamics in infinite width and depth networks. Investigate the proximity of finite network training dynamics to idealized limits through empirical analysis. Gain insights into a theoretical model of neural scaling laws that describes generalization dependence on training time, model size, and data quantity. Learn about compute-optimal scaling strategies, spectral properties of limiting kernels, and how representation learning can improve neural scaling laws. Understand the potential for doubling the training-time exponent in very hard tasks compared to the static kernel limit.