Modular Language Models: Addressing Limitations of Dense Training

Explore the concept of modular language models in this informative talk presented by Suchin Gururangan from the University of Washington. Delve into the limitations of conventional densely trained language models and discover a new class of fundamentally modular LMs. Learn how these modular models address issues such as prohibitive training costs and unreliable downstream behavior. Understand the benefits of modularity, including rapid customization, parallel processing, and sparse activation. Examine the importance of domain specialization for experts within the model and reflect on the data sources used for training. Gain insights into the potential for collaborative language model development, where contributions can be made with modest computational resources. This presentation, given at the USC Information Sciences Institute, offers valuable perspectives on the future of language model architecture and development.