Hugging Face Workshops - Pretraining Language Models & CodeParrot

Explore pretraining language models and Hugging Face's CodeParrot in this live workshop led by Leandro and Merve. Dive into the intricacies of data transformation, batching, and deep speed techniques. Learn about CodeParrot's capabilities for SQL and its evaluation process. Tackle coding challenges, address issues with duplicates, and understand deduplication methods. Gain insights into logging, model training loops, clipping, and checkpoints. Discover the potential of crosslingual transfer in natural language processing.