Explore a comprehensive video explanation of the Nyströmformer algorithm, a novel approach to approximating self-attention in Transformers with linear memory and time requirements. Delve into the quadratic memory bottleneck in self-attention, the softmax operation, and the Nyström approximation method. Gain insights into the landmark method, full algorithm implementation, theoretical guarantees, and techniques for avoiding large attention matrices. Compare subsampling keys with negative sampling, and examine experimental results demonstrating the algorithm's effectiveness. Enhance your understanding of this innovative solution for processing longer sequences in natural language processing tasks.
Nyströmformer- A Nyström-Based Algorithm for Approximating Self-Attention
Overview
Syllabus
- Intro & Overview
- The Quadratic Memory Bottleneck in Self-Attention
- The Softmax Operation in Attention
- Nyström-Approximation
- Getting Around the Softmax Problem
- Intuition for Landmark Method
- Full Algorithm
- Theoretical Guarantees
- Avoiding the Large Attention Matrix
- Subsampling Keys vs Negative Sampling
- Experimental Results
- Conclusion & Comments
Taught by
Yannic Kilcher
Related Courses
-
Rethinking Attention with Performers
-
Transformers Are RNNs- Fast Autoregressive Transformers With Linear Attention
-
Delta Keyword Transformer: Bringing Transformers to the Edge Through Dynamically Pruned Multi-Head Self-Attention
-
Big Bird- Transformers for Longer Sequences
-
Linformer - Self-Attention with Linear Complexity
-
Fast Multipole Attention: A Divide-and-Conquer Attention Mechanism for Long Sequences
