Learn about enhancing supernet architecture training through a 35-minute AutoML seminar that introduces the Mixture-of-Supernets formulation. Explore how Mixture-of-Experts (MoE) concepts generate flexible weights for subnetworks, leading to improved Neural Architecture Search (NAS) efficiency and high-quality architectures. Discover the practical applications in constructing efficient BERT and Machine Translation models while meeting user-defined constraints. Join speaker Ganesh Jawahar as he presents this ACL 2024 research that demonstrates significant improvements in retraining time and overall NAS effectiveness.
Mixture-of-Supernets - Improving Weight-Sharing Supernet Training with Architecture-Routed MoE
Overview
Syllabus
Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed MoE
Taught by
AutoML Seminars
Related Courses
-
Understanding Snowflake Arctic 480B - A Mixture of Experts LLM Architecture
-
Mixture of Experts (MoE) in Large Language Models - A Simple Guide
-
Stanford Seminar - Mixture of Experts Paradigm and the Switch Transformer
-
AutoMoE - Neural Architecture Search for Efficient Sparsely Activated Transformers
-
Mixture of Transformers for Multi-modal Foundation Models
-
Pioneering a Hybrid SSM Transformer Architecture - Jamba Foundation Model
