Data Sharding and the Art of Real-Time Analytics at Scale

Learn about implementing data sharding for real-time analytics at scale in this conference talk from DevOpsDays Tel Aviv. Explore how Singular tackled the challenge of achieving sub-second analytics queries while handling large volumes of real-time data using Apache Druid. Discover the journey from using a single real-time data source to implementing a custom data sharding mechanism, including the operational complexities involved in maintaining balanced shards, tuning stream and database configurations, and managing workload migration. Understand the limitations of batch-run optimizations and compaction tasks, and see how the sharding solution significantly improved query times, data storage optimization, and ingestion speed. Gain practical insights into the engineering and operational efforts required to build and maintain a high-performance real-time analytics system that effectively serves customer needs.