Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Learning Rate Grafting: Transferability of Optimizer Tuning - Machine Learning Research Paper Review

Yannic Kilcher via YouTube

Overview

Explore a comprehensive review of a machine learning research paper on Learning Rate Grafting, focusing on the transferability of optimizer tuning. Delve into the intricacies of various optimization algorithms like SGD, AdaGrad, Adam, LARS, and LAMB, examining their learning rate schedules and gradient direction corrections. Discover the grafting technique, which allows for transferring learning rate schedules between optimizers, and understand its implications for deep learning research. Learn about the experimental results, static transfer of learning rate ratios, and the potential for significant GPU memory savings. Gain insights into the entanglements between optimizers and learning rate schedules, and understand how grafting can provide a robust baseline for optimizer comparisons and reduce computational costs in hyperparameter searches.

Syllabus

- Rant about Reviewer #2
- Intro & Overview
- Adaptive Optimization Methods
- Grafting Algorithm
- Experimental Results
- Static Transfer of Learning Rate Ratios
- Conclusion & Discussion

Taught by

Yannic Kilcher

Reviews

Start your review of Learning Rate Grafting: Transferability of Optimizer Tuning - Machine Learning Research Paper Review

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.