Learn about the challenges and future improvements of Apache Pulsar's metrics system in this 24-minute technical talk from StreamNative's Asaf Mesika. Explore critical observability issues including topic count limitations, histogram visualization problems in Grafana, and data loss in rate and summary metrics. Discover practical solutions for monitoring large-scale Pulsar deployments, understand the limitations of current Prometheus export integration, and gain insights into proposed enhancements like Topic Metric Groups and OpenTelemetry adoption. Master the complexities of Pulsar Function Custom Metrics while learning about upcoming changes that will improve the overall monitoring experience for both users and system administrators.
Improving Observability Metrics in Apache Pulsar - Challenges and Solutions
StreamNative via YouTube
Overview
Syllabus
Intro
Monitoring 100k topics per broker
Hard to visualize Pulsar's histograms
Rate and Summary can lose monitoring data
Summary is not aggregatable
Pulsar Function Custom Metrics are Only Summary
Error prone and hard Prometheus Export integration
No pluggable way of adding another export format
New aggregation level: Topic Metric Group
Dynamic Filtering
Why OpenTelemetry?
What will change due to proposal?
Taught by
StreamNative
