Completed
Fine-tuning for long context and summarisation
Class Central Classrooms beta
YouTube videos curated by Class Central.
Classroom Contents
Advanced Fine-Tuning Techniques for Long Context Summarization
Automatically move to the next video in the Classroom when playback concludes
- 1 Fine-tuning for long context and summarisation
- 2 Video overview
- 3 Trick One: Increase rope_theta
- 4 Trick Two: Train norm and embed
- 5 Trick Three: Train on prompt + response
- 6 Long context and summarisation dataset
- 7 Fine-tuning script walk through
- 8 Prompt setup for summarisation
- 9 Raw Mistral 16k summarisation performance
- 10 Effect of increasing rope theta
- 11 Effect of training on summarization
- 12 64k context summarisation performance vs Yi 6B
- 13 Passkey retrieval performance of Mistral 64k
- 14 Resources
