Care and Feeding of Data Processing Pipelines

Learn about managing data processing pipelines in this 39-minute conference talk from SREcon18 Europe presented by Google's Rita Sodt. Explore essential strategies for handling pipelines that support critical functions like business analytics, machine learning, spam elimination, and billing processes. Discover why traditional monitoring approaches fall short for pipeline management and gain insights into preventing failures, implementing effective detection methods, and maintaining service level objectives (SLOs). Master specific techniques including canarying, incremental rollout, automatic failover, and auto-scaling tailored for pipeline environments. Understand the unique challenges of monitoring pipeline health through metrics like unprocessed data age and data correctness, rather than conventional RPC latency and error measurements.