What Learning Algorithm is In-Context Learning? - Understanding Transformer Models and Neural Sequence Learning

Watch a 50-minute lecture from MIT's Jacob Andreas at Harvard CMSA exploring the mechanisms behind in-context learning in neural sequence models, particularly transformers. Dive into research findings that demonstrate how transformer-based in-context learners implement standard learning algorithms by encoding smaller models in their activations and updating them as new examples appear in the context. Learn about the implementation of learning algorithms for linear models based on gradient descent and closed-form ridge regression, and discover how trained in-context learners match predictors computed by various regression methods. Explore the algorithmic features shared between in-context learners and these predictors, including how learners' late layers encode weight vectors and moment matrices. Follow along as the lecture covers topics from introduction to in-context learning, transformer models, offline transformations, learning setup, quality predictions, and future directions in this collaborative research effort between MIT, Stanford, and other institutions.