Dive deep into the metaseq codebase behind Meta's large language model OPT-175B in this comprehensive video tutorial. Learn how to set up the code on your machine and explore key concepts of mixed precision training, including loss scaling and unscaling. Follow along as the instructor walks through the training script, constructs dummy tasks and datasets, builds the transformer model, and examines CUDA kernels in C++ code. Gain insights into the training loop, forward pass through a transformer, and crucial aspects of loss handling, scaling, and mixed precision. Perfect for those looking to understand the intricacies of large language model implementation and training.
Overview
Syllabus
Intro - open pretrained transformer
Setup creating the cond env
Setup patch the code
Collecting train script arguments
Training script walk-through
Constructing a dummy task
Building the transformer model
CUDA kernels C++ code
Preparing a dummy dataset
Training loop
Zero grad loss scaling
Forward pass through a transformer
IMPORTANT loss, scaling, mixed precision, error handling
Outro
Taught by
Aleksa Gordić - The AI Epiphany
Related Courses
-
Create a Large Language Model from Scratch with Python – Tutorial
-
Ultimate Guide to Scaling ML Models - Megatron-LM - ZeRO - DeepSpeed - Mixed Precision
-
BigScience BLOOM - 3D Parallelism Explained - Large Language Models - ML Coding Series
-
Training and Testing an Italian BERT - Transformers From Scratch
-
8-bit Methods for Efficient Deep Learning
-
Distillation of Transformer Models - Tutorial and Code Walk-through
