Mandoline - Model Evaluation under Distribution Shift

This course focuses on teaching a new evaluation framework called Mandoline, which helps in model evaluation under distribution shift. The learning outcomes include understanding how to estimate model performance on a target distribution more accurately compared to standard baselines. The course teaches the concept of "slicing functions" and density ratio estimation for reweighted performance estimates. The teaching method involves theoretical explanations, empirical validations on NLP and vision tasks, and connecting the Mandoline framework to interactive ML systems. The intended audience for this course includes practitioners and researchers in the field of machine learning, particularly those interested in model evaluation and distribution shift challenges.