Sum-of-Minimum Model: Joint Optimization of Specialized Models for Heterogeneous Data

Explore a novel approach to machine learning optimization in this 26-minute conference talk from the "One World Optimization Seminar in Vienna" workshop. Dive into the "sum-of-minimum" model, a technique designed to handle heterogeneous data by jointly optimizing an ensemble of specialized models. Understand the mathematical formulation behind this approach, which aims to find the optimal assignment of data points to the best-performing models while simultaneously improving their performance. Learn about the challenges in solving this optimization problem, including non-smoothness and non-convexity issues. Discover an algorithm that approximately solves the problem, featuring an initialization step inspired by k-means++ and iterations similar to Lloyd's algorithm. Examine the performance and convergence bounds provided for this algorithm under certain assumptions. Explore practical applications of the "sum-of-minimum" model through experiments in generalized principal component analysis, neural network training, and mixed linear regression. Gain insights into how this innovative approach can potentially enhance machine learning performance when dealing with diverse and complex datasets.