Explore efficiency tricks for neural networks in natural language processing through this lecture from CMU's Neural Networks for NLP course. Delve into tips for training on GPUs, parallel training techniques, and softmax approximations including negative sampling and hierarchical softmax. Learn about the practical aspects of matrix-matrix multiplication, memory considerations, and three types of parallelism: within-operation, operation-wise, and example-wise. Discover how to implement data parallelism using modern libraries like PyTorch DistributedDataParallel. Examine methods for computation across large vocabularies, including noise contrastive estimation, mini-batch based negative sampling, class-based softmax, and binary code prediction. Gain insights into the glamorous life of an AI scientist and explore improvements to binary code prediction techniques.
Overview
Syllabus
Glamorous Life of an Al Scientist
A Simple Example • How long does a matrix-matrix multiply take?
Practically
What About Memory?
Three Types of Parallelism
Within-operation Parallelism
Operation-wise Parallelism
Example-wise Parallelism
Implementing Data Parallelism • Many modern libraries make data parallelism relatively easy, eg PyTorch DistributedDataParallel
Computation Across Large Vocabularies
Noise Contrastive Estimation (Mnih & Teh 2012)
Mini-batch Based Negative Sampling
Class-based Softmax (Goodman 2001) • Assign each word to a class • Predict class first, then word given class
Binary Code Prediction (Dietterich and Bakiri 1995, Oda et al. 2017)
Two Improvement to Binary Code Prediction
Taught by
Graham Neubig
Related Courses
-
Neural Nets for NLP - Efficiency Tricks for Neural Nets
-
Neural Nets for NLP - Structured Prediction with Local Independence Assumptions
-
Neural Nets for NLP - Class Introduction & Why Neural Nets?
-
CMU Neural Nets for NLP 2018 - Models of Words
-
Neural Nets for NLP 2019 - Word Vectors
-
Neural Nets for NLP 2020 - Language Modeling, Efficiency/Training Tricks
