Neural Nets for NLP 2019 - Unsupervised and Semi-supervised Learning of Structure

Overview

Explore a comprehensive lecture on unsupervised and semi-supervised learning of structure in neural networks for natural language processing. Delve into the differences between learning features and learning structure, examine various semi-supervised and unsupervised learning methods, and understand key design decisions for unsupervised models. Gain insights into practical examples of unsupervised learning, including hidden Markov models with Gaussian emissions, normalizing flow, and gated convolution. Investigate the challenges of learning latent structure and compare phrase structure to dependency structure. Learn about innovative approaches such as learning dependency heads with attention and reinforcement learning techniques for structure learning in NLP tasks.

Syllabus

Intro
Supervised, Unsupervised, Semi-supervised
Learning Features vs. Learning Discrete Structure
Unsupervised Feature Learning (Review)
How do we Use Learned Features?
What About Discrete Structure?
What is our Objective?
A Simple First Attempt
Hidden Markov Models w/ Gaussian Emissions . Instead of parameterizing each state with a categorical distribution, we can use a Gaussian (or Gaussian mixture)!
Problem: Embeddings May Not be Indicative of Syntax
Normalizing Flow (Rezende and Mohamed 2015)
Soft vs. Hard Tree Structure
One Other Paradigm: Weak Supervision
Gated Convolution (Cho et al. 2014)
Learning with RL (Yogatama et al. 2016)
Difficulties in Learning Latent Structure (Wiliams et al. 2018)
Phrase Structure vs. Dependency Structure
Learning Dependency Heads w/ Attention (Kuncoro et al. 2017)