Explore the process of fine-tuning Llama 3 for low-resource languages using Wikipedia datasets in this comprehensive 44-minute tutorial. Learn how to create a HuggingFace dataset using WikiExtractor, set up Llama 3 fine-tuning with LoRA, and implement dataset blending to prevent catastrophic forgetting. Dive into trainer setup, parameter selection, and loss inspection. Gain insights on learning rates, annealing, and additional tips for improving your fine-tuning results. Access provided resources including slides, dataset links, and code repositories to enhance your learning experience.
Fine-tuning Llama 3 on Wikipedia Datasets for Low-Resource Languages
Overview
Syllabus
Fine-tuning Llama 3 for a low resource language
Overview of Wikipedia Dataset and Loss Curves
Video overview
HuggingFace Dataset creation with WikiExtractor
Llama 3 fine-tuning setup, incl. LoRA
Dataset blending to avoid catastrophic forgetting
Trainer setup and parameter selection
Inspection of losses and results
Learning Rates and Annealing
Further tips and improvements
Taught by
Trelis Research
Related Courses
-
Fine-Tuning Llama 3 on a Custom Dataset for RAG Q&A - Training LLM on a Single GPU
-
Fine-Tuning Alpaca- Train Alpaca LoRa for Sentiment Analysis on a Custom Dataset
-
Fine-tuning Llama 2 for Tone or Style Using Shakespeare Dataset
-
Very Few Parameter Fine-Tuning with ReFT and LoRA
-
Preparing Fine-tuning Datasets with Open Source LLMs
-
LoRA Fine-tuning Explained - Choosing Parameters and Optimizations
