Knowledge Distillation Demystified: Techniques and Applications

Explore the powerful technique of knowledge distillation for optimizing machine learning models, particularly in natural language processing (NLP), in this 24-minute video presentation by Charlie Dickens, an applied research scientist at Snorkel AI. Learn about the fundamental concepts, benefits, methodologies, and real-world applications of knowledge distillation, which transfers knowledge from a large, complex model (the teacher) to a smaller, more efficient model (the student). Understand the two main steps of knowledge distillation: extraction and transfer. Discover how to identify target skills and curate seed knowledge for effective student model training. Examine various techniques for knowledge extraction, including teacher labeling, hidden representations, synthetic data, and feedback. Gain insight into the latest research and advancements in knowledge distillation, with a focus on the innovative data-centric approach being developed at Snorkel AI.