Modern Hopfield Networks for Novel Transformer Architectures

Explore a comprehensive seminar talk from Harvard CMSA's New Technologies in Mathematics series where IBM Research's Dmitry Krotov delves into Modern Hopfield Networks and their application in transformer architectures. Learn about the evolution from conventional Hopfield Networks of the 1980s to modern versions with significantly enhanced memory storage capacity. Discover the mathematical foundations and intuitive understanding of Dense Associative Memories as recurrent neural networks with fixed point attractor states. Examine the innovative Energy Transformer architecture, which integrates Dense Associative Memory models to replace traditional attention mechanisms. Understand the theoretical principles underlying this architectural advancement and review empirical results from its application in computer vision and graph network tasks. The presentation demonstrates how these modern neural network approaches are advancing machine learning, cognitive science, and neuroscience research.