μSlope - High Compression and Fast Search on Semi-Structured Logs

Explore a groundbreaking system for compressing and searching semi-structured log data in this 16-minute conference talk from OSDI '24. Dive into μSlope, a solution designed to address the challenges of managing massive amounts of log data produced by internet-scale services. Learn how this innovative approach achieves lossless compression of semi-structured formats like JSON while enabling fast search capabilities without full decompression. Discover the techniques used to concisely represent schema structures, "structurize" semi-structured data, and group records with similar schemas into well-structured tables. Examine the impressive compression ratios achieved by μSlope, ranging from 21.9:1 to 186.8:1, surpassing existing semi-structured data management systems. Gain insights into how this system outperforms Zstandard in compression and offers search speeds up to 5.77 times faster than other solutions. Understand the potential impact of μSlope on log data storage and analysis, particularly for companies like Uber that generate over 10PB of log data daily.