How Thanos Almost Snapped $100,000 from Our Infrastructure Budget

Explore a captivating conference talk that delves into the challenges of managing observability data in cloud-native environments. Learn how the implementation of Thanos and Fluentbit, powerful tools for metric storage and logging, nearly led to a $100,000 infrastructure budget overrun. Discover the potential pitfalls of configuring these tools, including easily overlooked caveats in data downsampling and log storage. Gain valuable insights from the speakers' experiences as they share their journey of implementing a Prometheus setup with high availability and virtually infinite historical data storage. Understand how Thanos Compactor's downsampling abilities can enhance query performance for older data, and how Fluentbit simplifies log collection and management across multiple sources and destinations. This talk offers crucial lessons for teams working with large-scale observability data, highlighting the importance of careful configuration and management of even the most powerful cloud-native tools.