Designing Load Balancer for Apache Pulsar Streaming Systems

Learn about Apache Pulsar's broker load balancer design and capabilities in this conference talk from Pulsar Summit NA 2023. Discover how traffic distribution is managed across Pulsar brokers through multiple settings including topic bundles, bundle assignment, bundle unload, and bundle split. Explore the new version of Broker Load Balancer based on Topics and Tableviews, including the introduction of Service Unit State and Load Data topics for broadcasting bundle ownership and load data. Understand how these topics reduce dependency on metadata store while addressing distributed data challenges like conflict resolution, consistency, recovery, and compaction. Gain comprehensive insights into Pulsar's broker-level auto-balancing mechanisms and how they ensure optimal traffic distribution across horizontally scalable messaging systems.