Shopify's Machine Learning Platform Journey: Building Merlin with Open Source Tools

Overview

Explore Shopify's journey in building Merlin, their new machine learning platform, in this 48-minute conference talk from MLOps World. Dive into the process, architecture, and impact of Merlin on scaling ML work at Shopify. Learn about the open-source stack and tooling used end-to-end, including Ray for distributed computing. Discover the platform's goals, timeline, and early vision. Gain insights into Merlin's architecture, API, workspaces, and user workflow. Understand the challenges of building and maintaining such a platform, including migration considerations. Follow along as Isaac Vidas, Machine Learning Platform Tech Lead at Shopify, shares valuable lessons and acknowledges the team's efforts in this comprehensive case study.

Syllabus

Intro
ML Platform goals
Background
Merlin Timeline
Merlin Early Vision
What is Ray?
Installing Ray
Ray Dashboard
Simple Ray Program
Tasks vs. Actors
Train Model with Ray XGBoost
ML Platform on Ray
Distributed Computation with Merlin Workspaces
Merlin Architecture
Merlin API
Merlin Workspace Details
Connecting to Merlin Workspaces
Isolating Merlin Workspaces
Structure of Merlin Project
Configuring a Merlin Project
Merlin CLI
Orchestration
User Workflow in Merlin
Balance
Building != Maintaining
Migration
What did we cover?
Acknowledgements