Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Scaling Transformer to 1M Tokens and Beyond with RMT - Paper Explained

Yannic Kilcher via YouTube

Overview

Explore a detailed analysis of the Recurrent Memory Transformer (RMT) technique, which promises to scale transformers to 1 million tokens and beyond. Learn about the strengths and weaknesses of this approach, its application to extend BERT's context length, and its potential impact on long-term dependency handling in natural language processing. Dive into the paper's key concepts, including the storage and processing of local and global information, information flow between input sequence segments, and experiments demonstrating the effectiveness of RMT. Gain insights into the unprecedented two-million-token context length achievement and its implications for memory-intensive applications in AI and language understanding.

Syllabus

- Intro
- Transformers on long sequences
- Tasks considered
- Recurrent Memory Transformer
- Experiments on scaling and attention maps
- Conclusion

Taught by

Yannic Kilcher

Reviews

Start your review of Scaling Transformer to 1M Tokens and Beyond with RMT - Paper Explained

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.