Learn about attention mechanisms in neural networks for natural language processing in this comprehensive lecture from CMU's Neural Networks for NLP course. Explore the "Attention is All You Need" paper, improvements to attention techniques, specialized attention varieties, and what neural networks actually attend to. Dive into topics like sentence representations, attention score functions, multi-headed attention, training tricks, and applications to various modalities. Gain insights on incorporating Markov properties, coverage, dictionary probabilities, and handling multiple sources in attention-based models.
Overview
Syllabus
Intro
Sentence Representations
Calculating Attention (1)
A Graphical Example
Attention Score Functions (1)
Attention Score Functions (2)
Multi-headed Attention
Attention Tricks
Summary of the Transformer
Training Tricks
Masking for Training
Incorporating Markov Properties
Coverage
Input Sentence: Copy
Dictionary Probabilities
Previously Generated Things
Various Modalities
Multiple Sources
Taught by
Graham Neubig
Related Courses
-
Neural Nets for NLP 2020 - Attention
-
Neural Nets for NLP 2017 - Attention
-
Neural Nets for NLP 2019 - Attention
-
CMU Multilingual NLP - Machine Translation-Sequence-to-Sequence Models
-
Neural Nets for NLP 2019: Sentence and Contextualized Word Representations
-
CMU Neural Nets for NLP - Model Interpretation
