Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Scalable MatMul-free Language Modeling - Paper Explained

Yannic Kilcher via YouTube

Overview

Explore a comprehensive video analysis of a research paper proposing MatMul-free language models. Delve into the innovative approach of replacing resource-intensive matrix multiplications with quantization and recurrence techniques to maintain performance while reducing computational costs. Learn about ternary accumulation as a substitute for matrix multiplication, the replacement of attention layers with recurrent layers, and the use of ternary channel mixing instead of dense layers. Examine the language modeling results, scaling laws, and other experimental outcomes presented in the paper. Gain insights into the potential for creating more efficient large language models and the implications for future hardware accelerators designed to process lightweight LLMs.

Syllabus

- Intro
- MatMul is everywhere
- Ternary accumulation as a substitute for matrix multiplication
- Replacing attention layers with recurrent layers
- Replacing dense layers with ternary channel mixing
- Language modelling results & scaling laws
- Other experimental results
- Conclusion

Taught by

Yannic Kilcher

Reviews

Start your review of Scalable MatMul-free Language Modeling - Paper Explained

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.