Explore advanced fine-tuning techniques in this comprehensive video tutorial on Combined Preference and Supervised Fine Tuning with ORPO. Learn about the evolution of fine-tuning methods, understand the differences between unsupervised, supervised, and preference-based approaches, and delve into cross-entropy and odds ratio loss functions. Discover why preference fine-tuning enhances performance through a hands-on notebook demonstration of SFT and ORPO. Evaluate the results using lm-evaluation-harness and compare SFT and ORPO performance across various benchmarks. Gain insights into the practical benefits of ORPO and access valuable resources for further exploration and implementation.
Combined Preference and Supervised Fine-Tuning with ORPO
Overview
Syllabus
Preference and Supervised Fine-tuning at the Same Time!
A short history of fine-tuning methods
Video Overview/Agenda
Difference between Unsupervised, Supervised and Preferences
Understanding cross-entropy and odds ratio loss functions
Why preference fine-tuning improves performance
Notebook demo of SFT and ORPO
Evaluation with lm-evaluation-harness
Results: Comparing SFT and ORPO with gsm8k, arithmetic and mmlu
Evaluation with Carlini's practical benchmark
Is it worth doing ORPO? Yes!
Taught by
Trelis Research
Related Courses
-
ORPO: Monolithic Preference Optimization without Reference Model
-
ORPO: A New Preference-Aligned Training Method for Large Language Models
-
Embeddings vs Fine-Tuning - Supervised Fine-tuning - Part 2
-
2025 Fine Tuning LLM with Hugging Face Transformers for NLP
-
Embeddings vs Fine-Tuning - Part 3: Unsupervised Fine-Tuning
-
FASTER Code for Supervised Fine-Tuning and DPO Training with UNSLOTH
