Explore word vectors in natural language processing through this comprehensive lecture from CMU's Neural Networks for NLP course. Dive into techniques for describing words by their context, including counting and prediction methods like skip-grams and CBOW. Learn how to evaluate and visualize word vectors, and discover advanced methods for creating more nuanced word representations. Examine the limitations of traditional embeddings and explore solutions like sub-word and multi-prototype embeddings. Gain insights into both intrinsic and extrinsic evaluation methods for word embeddings, and understand their practical applications in NLP systems.
Overview
Syllabus
Intro
What do we want to know about words?
A Manual Attempt: WordNet
An Answer (?): Word Embeddings!
Word Embeddings are Cool! (An Obligatory Slide)
How to Train Word Embeddings?
Distributional Representations (see Goldberg 10.4.1)
Count-based Methods
Word Embeddings from Language Models
Context Window Methods
Skip-gram (Mikolov et al. 2013) • Predict each word in the context given the word
Count-based and Prediction-based Methods
Glove (Pennington et al. 2014)
What Contexts?
Types of Evaluation
Extrinsic Evaluation: Using Word Embeddings in Systems
Intrinsic Evaluation of Embeddings (categorization from Schnabel et al 2015)
Limitations of Embeddings
Sub-word Embeddings (1)
Multi-prototype Embeddings • Simple idea, words with multiple meanings should have different embeddings (Reisinger and Mooney 2010)
Taught by
Graham Neubig
Related Courses
-
CMU Neural Nets for NLP - Distributional Semantics and Word Vectors
-
Neural Nets for NLP 2021 - Distributional Semantics and Word Vectors
-
CMU Neural Nets for NLP 2018 - Models of Words
-
Neural Nets for NLP 2017 - A Simple Exercise - Predicting the Next Word in a Sentence
-
Neural Nets for NLP 2021 - Neural Nets + Knowledge Bases
-
Neural Nets for NLP 2019: Sentence and Contextualized Word Representations
