Explore pretraining language models and Hugging Face's CodeParrot in this live workshop led by Leandro and Merve. Dive into the intricacies of data transformation, batching, and deep speed techniques. Learn about CodeParrot's capabilities for SQL and its evaluation process. Tackle coding challenges, address issues with duplicates, and understand deduplication methods. Gain insights into logging, model training loops, clipping, and checkpoints. Discover the potential of crosslingual transfer in natural language processing.
Overview
Syllabus
Intro
CodeParrot Overview
Pretraining Language Models
Data Transformation
CodeParrot
CoParrot
Batching
Iter
Tensor
Deep Speed
BigQuery vs DataSets
CodeParrot for SQL
Evaluation of CodeParrot
Coding Challenges
Problems with Duplicates
Deduplication
Questions
Logging
Models
Training Loop
Clipping
Checkpoints
More Questions
Crosslingual Transfer
Taught by
Hugging Face
