Caching Framework for Exabyte-Scale Data Lakes

Discover how to design and implement an open-source caching framework for exabyte-scale data lakes in this 40-minute conference talk. Learn about the challenges of data access performance and cost in large-scale data lakes, and explore solutions to improve performance by 1.5x while reducing storage costs by millions per year. Gain insights into architecting caching systems that accelerate queries, maximize cache hit rates, and cut data storage costs. Explore the open-source stack, including Hadoop, Parquet, Hudi, and Alluxio, used to achieve a balance between performance and cost efficiency. Delve into advanced techniques for improving cache hit rates, such as segmented data file caching, soft-affinity scheduler policies, and cache filtering. Learn how to monitor cache usage and working set size using comprehensive trace and JMX metrics. By the end of this session, acquire valuable knowledge to tackle the complexities of managing and optimizing exabyte-scale data lakes in both on-premises and cloud environments.