- Module 1: Learn the fundamental concepts of evaluating generative AI applications.
By the end of this module, you're able to:
- Apply best practices for choosing evaluation data
- Understood the purpose of and types of synthetic data for evaluation
- Comprehend the scope of the built-in metrics
- Choose the appropriate metrics based on your AI system use case
- Understand how to interpret evaluation results
- Module 2: Learn how to run evaluations and generate synthetic datasets with the Azure AI Evaluation SDK.
By the end of this module, you're able to:
- Assess a generative AI app response using performance and quality metrics
- Assess a generative AI app response using risk and safety metrics
- Run an evaluation and track the results in Azure AI Studio
- Create a custom evaluator with Prompty
- Send queries to an endpoint and run evaluators on the resulting query and response
- Generate a synthetic dataset using conversation starters
Overview
Syllabus
- Module 1: Module 1: Evaluating generative AI applications
- Introduction
- Evaluate generative AI
- The role of data in evaluations
- Choose and utilize metrics
- Custom prompt-based evaluators
- Interpret evaluation results
- Common pitfalls in result interpretation
- Knowledge check
- Summary
- Module 2: Module 2: Run evaluations and generate synthetic datasets
- Introduction
- Prepare
- Exercise - Performance and quality metrics
- Exercise - Risk and safety metrics
- Exercise - Track evaluation results in Azure AI Studio
- Exercise - Custom evaluator with Prompty
- Exercise - Evaluate an endpoint
- Exercise - Generate a synthetic dataset
- Knowledge check
- Summary
