Overview
Syllabus
Intro
Nearline Applications
Heterogeneous Data Systems
Building the Right Infrastructure
Pluggable Sources & Destinations
Capturing Live Updates
Change Data Capture (CDC)
Streaming Bridge
Mirroring Kalka Data
Brooklin Kalka Mirroring
Application Use Cases
Client makes REST call to create datastream
Datastream is written to Zookeeper
Leader coordinator is notified of new datastream
Leader coordinator calculates work distribution
Leader coordinator writes the assignments to ZK
Zookeeper is used to communicate the assignments
Coordinators hand task assignments to consumers
Consumers start streaming data from the source
Consumers propagate data to producers
Producers write data to the destination
Destinations can be shared by apps
Brooklin Architecture
Brooklin in Production
Brooklin is now open-source!
Taught by
Strange Loop Conference