Towards Better Understanding of Contrastive Learning

Overview

Explore a comprehensive presentation on contrastive learning delivered by Yuandong Tian from Meta for the Data Learning working group. Delve into the empirical success of deep models, self-supervised learning, and the formulation of contrastive learning. Examine the understanding of contrastive loss, the InfoNCE example, and coordinate-wise optimization. Discover a surprising connection to kernels and gain insights into nonlinear analysis. Investigate training dynamics, including 1-layer 1-node and multiple node nonlinear networks. Learn about conditional independence, global modulation, and feature emergence. Compare quadratic loss versus InfoNCE through experimental settings, model architecture, and evaluation metrics. This 58-minute talk, recorded on March 7, 2023, offers valuable insights for researchers and students developing new technologies based on Data Assimilation and Machine Learning.

Syllabus

Intro
Great Empirical Success of Deep Models
Self-supervised Learning (SSL)
Contrastive Learning (CL)
Formulation of Contrastive Learning
Understanding of Contrastive Loss
What Deep Learning Brings?
Example: InfoNCE
Coordinate-wise Optimization
A Surprising Connection to Kernels
Overview of Nonlinear Analysis
Nonlinear Setting
Training Dynamics
1-layer 1-node nonlinear network
How to reduce the local roughness p(w)?
1-layer multiple node nonlinear network
Assumptions
Conditional Independence
What linear network cannot do
Global modulation
Feature Emergence
Experiment Setting
Model Architecture & Evaluation Metric
Visualization
Quadratic Loss versus InfoNCE