Learn how to set up and use the Llama 2 tokenizer in this 15-minute tutorial video from Trelis Research. Explore adding BOS and EOS tokens, setting pad and mask tokens, and understanding the prompt format for single and multi-turn interactions with Llama 2. Access the accompanying GitHub repository and Colab notebook for hands-on practice with the concepts covered.
Llama 2 Tokenizer, Padding, and Prompt Format Tutorial
Overview
Syllabus
Introduction and colab notebook setup for Llama 2
How to add BOS and EOS tokens with the Llama 2 tokenizer?
How to set a pad token for Llama 2?
How to set a mask token for Llama 2?
What is the prompt format for Llama 2?
What is the multi-turn prompt format for Llama 2?
Finding the Github Repo and Colab Notebook Tutorial
Taught by
Trelis Research
