Maximizing Query Speed and Minimizing Costs in Data Lakes with Open-Source Caching

Learn how to optimize data lake performance and reduce costs through open-source caching in this 26-minute conference talk from OSA Con 2023. Discover architectural approaches for implementing caching frameworks that leverage tools like Hadoop, Parquet, Hudi, and Alluxio to achieve 1.5x faster performance and significant cost savings in both on-premises and cloud environments. Explore key challenges in large-scale data lakes, techniques for maximizing cache hit rates, strategies for efficient storage utilization, and methods for implementing advanced features like segmented file caching and affinity-based scheduling. Master the implementation of cache monitoring systems using metrics and traces while gaining practical insights into optimizing cache usage through filtering and working set analysis.