Completed
Absolute Position Encoding and Relative Position Encoding (RPE)
Class Central Classrooms beta
YouTube videos curated by Class Central.
Classroom Contents
Rethinking and Improving Relative Position Encoding for Vision Transformer - Lecture 23
Automatically move to the next video in the Classroom when playback concludes
- 1 Intro
- 2 Background and previous work
- 3 Self-attention
- 4 Absolute Position Encoding and Relative Position Encoding (RPE)
- 5 RPE in Transformer-XL
- 6 Bias and Contextual Mode
- 7 A Piecewise Index Function
- 8 2D Relative Position Calculation
- 9 Experiments
- 10 Implementation details
- 11 Directed vs. Undirected Bias vs. Contextual
- 12 Shared v.s. Unshared
- 13 Piecewise v.s. Clip
- 14 Number of buckets
- 15 Component-wise analysis
- 16 Complexity Analysis
- 17 Visualization
- 18 Conclusion
