Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

Stanford University

Demystifying Mixtral of Experts - Stanford CS25 Lecture

Stanford University via YouTube

Overview

Explore the intricacies of Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model, in this illuminating lecture from Stanford's CS25 course. Delve into the architectural details of Mixtral, which builds upon the Mistral 7B framework but incorporates 8 feedforward blocks (experts) in each layer. Discover how the model's router network selects and combines outputs from two experts per token at each layer, allowing access to 47B parameters while actively using only 13B during inference. Gain insights into the expert routing decisions and their implications. Presented by Albert Jiang, an AI scientist at Mistral AI and PhD student at Cambridge University, this talk offers a deep dive into cutting-edge language model architecture and its applications in pretraining and reasoning.

Syllabus

Stanford CS25: V4 I Demystifying Mixtral of Experts

Taught by

Stanford Online

Reviews

Start your review of Demystifying Mixtral of Experts - Stanford CS25 Lecture

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.