Explore a comprehensive 55-minute conference talk on testing machine learning models in production. Learn about core statistical tests and metrics for detecting data and concept drift, preventing models from becoming stale and detrimental to business. Dive deep into implementing robust testing and monitoring frameworks using open-source tools like MLflow, SciPy, and statsmodels. Gain valuable insights from Databricks' customer experiences and discover key tenets for testing model and data validity in production. Walk through a generalizable demo utilizing MLflow to enhance reproducibility. Cover topics including the ML cycle, data monitoring, KS tests, categorical features, one-way chi-squared tests, monitoring tools, MLflow notebooks, ML workflows, data logging, model registry, feature checks, and model staging and migration.
Overview
Syllabus
Intro
ML Cycle
Data Monitoring
KS Test
Categorical Features
Oneway Chisquared
Monitoring Tests
Tools
ML Flow
Notebooks
ML Workflow
ML Flow Delta
Other Notebooks
Widgets
Notebook Setup
Train Cycle Learn Pipeline
Data Logging
ML Flow Run
ML Flow Model Registry
ML Flow Experiment
Model Registry
New Data
Feature Checks
ML Flow Registry
Null Proportion
New Incoming Data
Chisquared Test
Action
Model Parameters
Model Staging
Model Migration
Missingness Check
Price Check
Categorical Check
Recap
Taught by
Databricks
Related Courses
-
Monitoring ML Models - Full Stack Deep Learning - Spring 2021
-
Keeping Up With ML Models in Production: Mitigating Performance Drift
-
Evidently AI Tutorial - Open Source ML Models Monitoring and Observability
-
Building and Maintaining High-Performance AI
-
How Not to Let Your Data and Model Drift Away Silently
-
Introduction to Drifter ML: A Testing Framework for Machine Learning Models
