Explore a comprehensive podcast episode featuring Gideon Mendels, CEO of Comet, discussing systematic testing and evaluation of LLM applications. Gain insights into hybrid approaches combining ML and software engineering best practices, defining evaluation metrics, and tracking experimentation for LLM app development. Learn about comprehensive unit testing strategies for confident deployment, and discover the importance of managing machine learning workflows from experimentation to production. Delve into topics such as LLM evaluation methodologies, AI metrics integration, experiment tracking, collaborative approaches, and anomaly detection in model outputs. Benefit from Mendels' expertise in NLP, speech recognition, and ML research as he shares valuable insights for developers working with LLM applications.
How to Systematically Test and Evaluate LLM Apps - MLOps Podcast
Overview
Syllabus
[] Gideon's preferred coffee
[] Takeaways
[] A huge shout-out to Comet ML for sponsoring this episode!
[] Please like, share, leave a review, and subscribe to our MLOps channels!
[] Evaluation metrics in AI
[] LLM Evaluation in Practice
[] LLM testing methodologies
[] LLM as a judge
[] OPIC track function overview
[] Tracking user response value
[] Exploring AI metrics integration
[] Experiment tracking and LLMs
[] Micro Macro collaboration in AI
[] RAG Pipeline Reproducibility Snapshot
[] Collaborative experiment tracking
[] Feature flags in CI/CD
[] Labeling challenges and solutions
[] LLM output quality alerts
[] Anomaly detection in model outputs
[] Wrap up
Taught by
MLOps.community
Related Courses
-
Reliable LLM Products Fueled by Feedback - MLOps Podcast #251
-
Enterprises Using MLOps - The Changing LLM Landscape and MLOps Pipelines
-
MLOps at the Crossroads - Challenges and Future Directions
-
LLM Evaluation: Challenges and Best Practices - MLOps Podcast #210
-
Experiment Tracking in the Age of LLMs - MLOps Podcast
-
Managing Data for Effective GenAI Application - MLOps Podcast #216
