Emergence and Grokking in Simple Neural Architectures

Explore a thought-provoking lecture on the emergence and grokking phenomena in simple neural network architectures. Delve into Misha Belkin's presentation at IPAM's Theory and Practice of Deep Learning Workshop, where he argues that Multi-Layer Perceptrons (MLPs) exhibit remarkable behaviors similar to those observed in modern Large Language Models. Examine the challenges in understanding how 2-layer MLPs learn relatively simple problems like "grokking" modular arithmetic. Discover recent progress in the field and learn about Recursive Feature Machines as a potential model for analyzing emergent phenomena in neural architectures. Gain valuable insights into the computational aspects of modern neural networks and their implications for deep learning theory and practice.