KalDB - A Cloud Native Log Search Platform

Explore the development of KalDB, a cloud-native log search platform designed to address challenges faced by Slack's Elasticsearch-based system. Learn about the innovative architecture that prioritizes real-time log visibility, automatically handles field conflicts, and offers first-class Kubernetes support. Discover how this solution tackles issues of delayed logs, ingestion failures, and operational complexity while employing techniques like S3-backed storage to reduce infrastructure costs. Gain insights into the motivations behind KalDB's creation, its multitenancy capabilities, and the significant cost savings it provides. Presented by Suman Karumuri, Sr. Staff Software Engineer at Slack and expert in distributed tracing, this conference talk covers the journey from identifying problems with the existing system to implementing a more efficient and scalable log search platform.