Advanced Fine-Tuning Techniques for Long Context Summarization

Advanced Fine-Tuning Techniques for Long Context Summarization

Trelis Research via YouTube Direct link

Resources

14 of 14

14 of 14

Resources

Class Central Classrooms beta

YouTube videos curated by Class Central.

Classroom Contents

Advanced Fine-Tuning Techniques for Long Context Summarization

Automatically move to the next video in the Classroom when playback concludes

  1. 1 Fine-tuning for long context and summarisation
  2. 2 Video overview
  3. 3 Trick One: Increase rope_theta
  4. 4 Trick Two: Train norm and embed
  5. 5 Trick Three: Train on prompt + response
  6. 6 Long context and summarisation dataset
  7. 7 Fine-tuning script walk through
  8. 8 Prompt setup for summarisation
  9. 9 Raw Mistral 16k summarisation performance
  10. 10 Effect of increasing rope theta
  11. 11 Effect of training on summarization
  12. 12 64k context summarisation performance vs Yi 6B
  13. 13 Passkey retrieval performance of Mistral 64k
  14. 14 Resources

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.