Overview
Syllabus
Intro
ML Research Pipeline
Concern #1: "Classic" Overfitting
Concern #2: Adaptive Overfitting
Simple Setting: Background bias
Do Backgrounds Contain Signal?
ImageNet-9: A Fine-Grained Study Xiao Engstrom Ilyas M 2020
Adversarial Backgrounds
Background-Robust Models?
Are Better Models Better?
Biases Can Be Subtle
How Are Datasets Created?
Dataset Creation in Practice
Crowdsourced Validation: A Closer Look
Prerequisite: Detailed Annotations
Restricting Relevant Labels
From Validation to Classification
Multi-Object Images
How Does This Affect Accuracy?
Which Object Do Models Predict?
Human-Based Evaluation
Dataset Replication
Case Study: ImageNet-v2
Replication Pipeline
Statistical Bias
Taught by
Institute for Advanced Study