Delta Keyword Transformer: Bringing Transformers to the Edge Through Dynamically Pruned Multi-Head Self-Attention

Explore the cutting-edge developments in bringing Transformers to edge devices through a 21-minute conference talk from the tinyML Research Symposium 2022. Delve into the innovative Delta Keyword Transformer, presented by Zuzana Jelčicoová, an Industrial PhD student at Oticon. Learn about dynamically pruned multi-head self-attention and its applications in edge computing. Gain insights into the Keyword Transformer (KWT) model analysis, the Delta algorithm, and its implementations in regular and delta matrix multiplication, as well as softmax operations. Discover the results and implications of this groundbreaking research, concluding with a glimpse into EDGE IMPULSE technology.