Resilient Predictive Data Pipelines

Explore resilient predictive data pipelines in this 45-minute conference talk from GOTO Chicago 2016. Dive into the world of big data architecture with Siddharth "Sid" Anand, Data Architect at Agari Inc. Learn how major tech companies build custom data pipelines to meet strict requirements on security, fault-tolerance, cost control, scalability, and uptime. Discover key concepts such as data products, serving pipelines, and the blast radius problem. Gain insights into timeliness, dead letter queues, and the SNS + SOS design pattern. Explore Apache Airflow for authoring DAGs, performance insights, and alerting. Examine near-real-time architecture, schema registry, AWS Lambda, and elastic stream processing. Perfect for data engineers and architects looking to enhance their knowledge of resilient and scalable data pipeline design.