Explore the fundamentals of word representations in natural language processing through this comprehensive lecture from Carnegie Mellon University's Neural Networks for NLP course. Delve into various approaches for modeling words, from manual attempts like WordNet to modern word embedding techniques. Examine the differences between distributional and distributed representations, and learn about count-based and prediction-based methods for training word embeddings. Investigate the importance of context in word representations and discover different evaluation techniques for assessing embedding quality. Analyze the strengths and limitations of word embeddings, and gain insights into choosing appropriate embeddings for specific tasks. Conclude by exploring sub-word embedding techniques to address limitations of traditional word-level representations.
Overview
Syllabus
Intro
What do we want to know about words?
A Manual Attempt: WordNet
An Answer (?): Word Embeddings!
How to Train Word Embeddings?
Distributional vs. Distributed Representations
Count-based Methods
Distributional Representations (see Goldberg 10.4.1) • Words appear in a context
Context Window Methods
Count-based and Prediction-based Methods
Glove (Pennington et al. 2014)
What Contexts?
Types of Evaluation
Non-linear Projection • Non-linear projections group things that are close in high- dimensional space eg. SNEA-SNE (van der Masten and Hinton 2008) group things that give each other a high probability according to a Gaussian
t-SNE Visualization can be Misleading! (Wattenberg et al. 2016)
Intrinsic Evaluation of Embeddings (categorization from Schnabel et al 2015)
Extrinsic Evaluation: Using Word Embeddings in Systems
How Do I Choose Embeddings?
When are Pre-trained Embeddings Useful?
Limitations of Embeddings
Sub-word Embeddings (1)
Taught by
Graham Neubig
Related Courses
-
CMU Neural Nets for NLP - Distributional Semantics and Word Vectors
-
Neural Nets for NLP 2021 - Distributional Semantics and Word Vectors
-
Neural Nets for NLP 2019 - Word Vectors
-
Neural Nets for NLP 2021 - Conditioned Generation
-
Neural Nets for NLP 2021 - Neural Nets + Knowledge Bases
-
Neural Nets for NLP 2018 - Learning from-for Knowledge Graphs
