Dive into a comprehensive two-part conference talk from SREcon19 Asia/Pacific that explores the essential building blocks of distributed systems. Learn about orchestration, load balancing, databases, and storage services through the lens of designing a large data processing pipeline. Engage in an interactive session where John Looney from Facebook discusses crucial concepts, including pipeline and batch systems, failover, lockservers, consensus algorithms, queues, data storage, and database types. Participate in a collaborative design review of a theoretical pipeline system to solidify your understanding of these complex topics. Gain valuable insights into the challenges and solutions in distributed systems architecture, from CAP theorem to cluster filesystems, in this comprehensive 1 hour and 23 minutes presentation.
Overview
Syllabus
Intro
Pipeline & Batch Systems (Part 1)
Orchestration: Finding, Ordering, Sharding
Making Reliability Worse: Failover
Lockservers, discovery
Clients, self-resolution
So, what lockserver ?
Let's talk about Consensus
Consensus Challenges
Consensus; Requirements
Consensus; Raft
Spot the Difference!
Ordered Queues: Pain And Suffering
Queues: PubSub & SQS
Queues: Kafka, Log Device, Kinesis
Data Storage: CAP Theorem
Data Storage; B-Trees vs. LSM
Data Storage: Weak vs Strong Isolation
Data Storage: Database Types
Datacenter / Cluster Filesystems
Useful Distributed DB/Cluster patterns
Taught by
USENIX
