Explore advanced techniques for modeling long sequences in natural language processing through this comprehensive lecture from CMU's Advanced NLP course. Delve into extracting features from extended text and tackling document processing tasks. Learn about various transformer architectures including Transformer XL, Compressive Transformers, and Sparse Transformers. Examine adaptive span and sparse span approaches, as well as the Reformer model. Investigate low rank approximation and sparse attention methods. Gain insights into evaluation techniques and other relevant methodologies. Conclude with an overview of coreference models, including mention pair models and their components.
Overview
Syllabus
Introduction
NLP Tasks
Modeling Long Sequences
Separate Encoding
Selfattention Transformers
Transformer XL
Compressive Transformers
Sparse Transformers
Adaptive Span Transformers
Sparse Span Transformers
Reformer Model
Low Rank Approximation
Sparse Attention
Evaluation
Other Methods
Questions
Components of Coreference Models
Mention Pair Models
Model
Taught by
Graham Neubig
