Learn how to build and train a custom tokenizer for use with Transformers in this 23-minute tutorial. Explore the fundamentals of tokenizers, understand when to train a custom tokenizer, and discover the importance of special tokens. Follow along with Lucile, a machine learning engineer at Hugging Face, as she guides you through the process using Transformer Notebooks. Gain valuable insights into Natural Language Processing and enhance your skills in developing open-source tools for collaborative training and research projects.
Overview
Syllabus
Introduction
What are tokenizers
Transformer Notebooks
Training a tokenizer
When should I train a tokenizer
What is a tokenizer
Special tokens
Taught by
Hugging Face
