AWS ML Engineer Associate 2.4 Analyze Model Performance

This final course of the model development domain provides instructions to analyze ML model performance. You will learn about key concepts and techniques for model evaluation including classification and regression problem metrics. You will also learn how to identify convergence issues and ensure reproducible experimentation. Finally, you will use AWS services such as Amazon SageMaker Clarify and Amazon SageMaker Debugger to gain insight into machine learning (ML) training data and model issues.

• Course level: Advanced
• Duration: 1.5 hours

Activities

• Online materials
• Exercises
• Knowledge check questions

Course objectives

• Determine methods for creating performance baselines.
• Assess trade-offs between model performance, training time, and cost.
• Determine classification problem evaluation techniques and metrics.
• Determine regression problem evaluation techniques and metrics.
• Identify convergence issues and prevent model convergence issues with Amazon SageMaker Training Compiler and Amazon SageMaker Automatic Model Tuning (AMT).
• Identify SageMaker Clarify metrics for gaining insights into ML training data and models.
• Use SageMaker Clarify to interpret model outputs.
• Describe how to perform reproducible experiments using AWS services. 
• Use SageMaker Model Debugger to debug model convergence.

Intended audience

• Cloud architects
• Machine learning engineers

Recommended Skills

• Completed at least 1 year of experience using SageMaker and other AWS services for ML engineering
• Completed at least 1 year of experience in a related role such as backend software developer, DevOps developer, data engineer, or data scientist
• A fundamental understanding of programming languages such as Python
• Completed preceding courses in the AWS ML Engineer Associate Learning Plan

Course outline

• Section 1: Introduction
  • Lesson 1: How to Use This Course
  • Lesson 2: Course Overview
  • Lesson 3: Performance Baselines
• Section 2: Model Evaluation
  • Lesson 4: Model Evaluation Techniques and Metrics
  • Lesson 5: Convergence Issues
  • Lesson 6: Debug Model Convergence with SageMaker Debugger 
  • Lesson 7: SageMaker Clarify and Metrics Overview
  • Lesson 8: Interpret Model Outputs Using SageMaker Clarify
  • Lesson 9: Amazon SageMaker Experiments
• Section 3: Conclusion
  • Lesson 10: Course Summary
  • Lesson 11: Assessment
  • Lesson 12: Contact Us