Trusting Your Data Science Pipelines - A Quantitative Approach

Explore a comprehensive approach to ensuring reliability in complex data science pipelines through this conference talk from Strange Loop. Delve into the challenges faced by companies developing systems that transform massive raw datasets into actionable signals using advanced machine learning techniques and distributed computing. Learn about the concept of Quantitative Reliability Engineering (QRE) and how it addresses the issue of technical myopia in specialized teams. Discover the scientific methodology employed by QRE to understand and enhance every step of a data analytics pipeline. Gain insights into the collaboration between QRE teams, data scientists, and engineers to verify end-to-end results and develop scalable algorithms. Join Elijah ben Izzy, the tech lead of Two Sigma's QRE team, as he shares their experiences, challenges, and how their approach applies to the broader computational industry in this 39-minute presentation.