Why AI Needs New Data Benchmarks and Quality Metrics

Explore the critical need for new data benchmarks and quality metrics in AI through this 27-minute talk from Snorkel AI. Delve into the challenges arising from the past decade's focus on ML models and prominent open-source datasets, which has led to widespread issues like data cascades in real-world applications and the saturation of existing dataset-driven benchmarks. Learn about DataPerf, a benchmark suite designed to evaluate training and test data quality, as well as algorithms for dataset construction and optimization across various ML tasks. Discover how DataPerf, supported by the MLCommons Association, aims to stimulate research in data-centric AI and promote data excellence. Gain insights into this collaborative effort involving multiple organizations, including Coactive.AI, ETH Zurich, Google, Harvard University, Landing.AI, Meta, Stanford University, and TU Eindhoven. Understand the importance of community participation in defining industry benchmarks for ML datasets and how this initiative can drive advancements in AI research and development.