Learn about an innovative approach to Large Language Model training in this 24-minute technical presentation that introduces ORPO (Odds Ratio Preference Optimization), a groundbreaking "reference model-free" monolithic optimization algorithm. Explore the theoretical physics perspective behind this new preference-aligned Supervised Fine-Tuning (SFT) method, examining parallels between regularization terms methodologies and Lagrange Multipliers. Delve into how ORPO eliminates the need for a separate preference alignment phase while comparing its performance metrics against LLama 2 and Mistral 7B models. Based on research from the paper "ORPO: Monolithic Preference Optimization without Reference Model," gain insights into this streamlined approach that combines preference alignment directly into the training process.
Overview
Syllabus
ORPO: NEW DPO Alignment and SFT Method for LLM
Taught by
Discover AI
Related Courses
-
ORPO: Monolithic Preference Optimization without Reference Model
-
FASTER Code for Supervised Fine-Tuning and DPO Training with UNSLOTH
-
Combined Preference and Supervised Fine-Tuning with ORPO
-
Direct Preference Optimization (DPO) vs RLHF - Understanding Language Model Training
-
Aligning Language Models with LESS Data and Simple Preference Optimization (SimPO)
-
Fine-Tuning Self-Rewarding Language Models with Mistral 7B
