Learn about distributional semantics and word vectors in this comprehensive lecture from CMU's Neural Networks for NLP course. Explore techniques for describing words by their context, including counting and prediction methods. Dive into skip-grams, continuous bag-of-words (CBOW), and advanced word vector approaches. Discover methods for evaluating and visualizing word vectors, and gain insights into their limitations and applications. Examine topics such as contextualization, WordNet, GloVe, intrinsic and extrinsic evaluation, pre-trained embeddings, and techniques for improving embeddings like retrofitting and de-biasing.
Overview
Syllabus
Intro
Remember: Neural Models
How to Train Embeddings?
What do we want to know about words?
Contextualization of Word Representations
A Manual Attempt: WordNet
An Answer (?): Word Embeddings!
Word Embeddings are Cool! (An Obligatory Slide)
Distributional vs. Distributed Representations
Distributional Representations (see Goldberg 10.4.1)
Count-based Methods
Prediction-basd Methods (See Goldberg 10.4.2)
Word Embeddings from Language Models giving
Context Window Methods
Glove (Pennington et al. 2014)
What Contexts?
Types of Evaluation
Non-linear Projection • Non-linear projections group things that are close in high
t-SNE Visualization can be Misleading! Wattenberg et al. 2016
Intrinsic Evaluation of Embeddings (categorization from Schnabel et al 2015)
Extrinsic Evaluation
How Do I Choose Embeddings?
When are Pre-trained Embeddings Useful?
Limitations of Embeddings
Unsupervised Coordination of Embeddings
Retrofitting of Embeddings to Existing Lexicons . We have an existing lexicon like WordNet, and would like our vectors to match (Faruqui et al. 2015)
Sparse Embeddings
De-biasing Word
FastText Toolkit
Taught by
Graham Neubig
Related Courses
-
CMU Neural Nets for NLP - Distributional Semantics and Word Vectors
-
Neural Nets for NLP 2019 - Word Vectors
-
CMU Neural Nets for NLP 2018 - Models of Words
-
Neural Nets for NLP 2017 - A Simple Exercise - Predicting the Next Word in a Sentence
-
Neural Nets for NLP 2021 - Neural Nets + Knowledge Bases
-
Neural Nets for NLP 2018 - Learning from-for Knowledge Graphs
