In this course, you will learn techniques for monitoring and maintaining the performance and reliability of your machine learning (ML) solutions using the monitoring capabilities of Amazon SageMaker. You begin by establishing the importance of monitoring and the types of drift in ML. Then, you will discover methods to detect data drift, model quality issues, statistical bias, and feature attribution drift. You will explore SageMaker Model Monitor for continuous monitoring, SageMaker Clarify for detecting bias and providing interpretable explanations, and SageMaker Model Dashboard for visualizing and analyzing performance metrics.
This course shares best practices to help you build and maintain reliable, high-performing, and trustworthy ML solutions that align with the AWS Well-Architected Machine Learning Lens design principles. You will learn approaches for proactive decision-making, automated remediation, notifications, and retraining workflows, which will help keep your ML solutions effective over time.
- Course level: Advanced
- Duration: 2 hours and 30 minutes
Activities
- Online materials
- Exercises
- Knowledge check questions
Course objectives
- Describe the AWS Well-Architected Machine Learning Lens design principles for monitoring.
- Identify best practices to monitor data quality and model performance.
- Use SageMaker Model Monitor to continuously monitor models in production for data drift and model quality issues.
- Explain how Amazon SageMaker Clarify can detect model bias and provide interpretable explanations.
- Describe the benefits and use cases of SageMaker Clarify for attribution monitoring.
- Describe the benefits of monitoring model performance in production using A/B testing.
- Explain the key features and common use cases of SageMaker Model Dashboard.
- Proactively identify issues by monitoring ML solutions and implementing automated remediation, notifications, and retraining workflows.
Intended audience
- Cloud architects
- Machine learning engineers
Recommended Skills
- Completed at least 1 year of experience using SageMaker and other AWS services for ML engineering
- Completed at least 1 year of experience in a related role, such as backend software developer, DevOps developer, data engineer, or data scientist
- A fundamental understanding of programming languages, such as Python
- Completed preceding courses in the AWS ML Engineer Associate Learning Plan
Course outline
- Section 1: Introduction
- Lesson 1: How to Use This Course
- Lesson 2: Domain Introduction
- Lesson 3: Course Overview
- Section 2: Monitoring Machine Learning Solutions
- Lesson 4: The Importance of Monitoring in ML
- Lesson 5: Detecting Drift in Monitoring
- Lesson 6: Amazon SageMaker Model Monitor
- Lesson 7: Monitoring for Data Quality Drift
- Lesson 8: Monitoring for Model Quality Using SageMaker Model Monitor
- Lesson 9: SageMaker Model Monitor Demo
- Lesson 10: Monitoring for Statistical Bias Drift with SageMaker Clarify
- Lesson 11: Monitoring for Feature Attribution Drift
- Lesson 12: Monitoring Model Performance Using A/B Testing
- Lesson 13: Introduction to SageMaker Model Dashboard
- Lesson 14: Choosing Your Monitoring Approach
- Section 3: Remediating Problems Identified by Monitoring
- Lesson 15: Automated Remediation and Troubleshooting
- Section 4: Conclusion
- Lesson 16: Course Summary
- Lesson 17: Assessment
- Lesson 18: Contact Us
Keywords
- Gen AI
- Generative AI
