Learn how DoorDash scaled their event processing system to handle hundreds of billions of events in this technical conference talk from InfoQ Dev Summit. Explore the architecture and implementation details of a large-scale real-time event processing system, covering event production optimization, Kafka Rest Proxy utilization, and Flink integration. Dive into practical solutions for challenges like schema validation, data warehouse integration, and building self-serve platforms. Discover how to leverage FlinkSQL for feature engineering, implement unified event formats, and create abstractions for better scalability. Master key concepts in event-driven architecture including batching strategies, layered APIs, and fine-grained failure isolation while following DoorDash's journey from zero to processing hundreds of billions of events efficiently.
Building Scalable Real-Time Event Processing Systems - From Zero to 100 Billion Events
Overview
Syllabus
Intro
Successful Patterns in Building Realtime Event Processing
Realtime Events Use Case at DoorDash
Requirements for the New System
The Big Picture
Simplify and optimize event producing
Leveraging Kafka Rest Proxy
Batching with Kafka Rest Proxy
Our Proxy Enhancements
Event Processing with Flink
Challenges of a simple Kafka consumer
Leveraging Flink's layered APIs
Flink Platform at DoorDash
Riviera - Applying FlinkSQL in Feature Engineering
Riviera DSL Example For Store Order Count
Unified Event Format and API to Reduce Frictions
Leveraging Schema Registry for Generic Data Processing
Schema update strategy
Build time schema update automation
Data Warehouse and Data Lake Integration
Working Towards a Self-serve Platform
Snowflake Integration Management UI
Leveraging Stream Processing Framework
Creating Abstractions
Fine-grained Failure Isolation and Scalability
Beyond the Architecture
Taught by
InfoQ
