Rapid PySpark Custom Processing on Time Series Big Data in Databricks

Discover how Sleep Number leveraged Pyspark and Databricks to efficiently process massive time series data from smartbed sensors in this 28-minute conference talk. Learn about the challenges of analyzing noisy sensor readings and the implementation of custom entropy calculations on rolling windows. Explore the transition from a memory-constrained Pandas approach to a scalable Pyspark solution that processed 50 million records in just 0.3 seconds. Gain insights into optimizing big data processing for constant time complexity, regardless of data size. Presented by Gary Garcia Molina and Megha Rajam Rao from Sleep Number, this talk demonstrates advanced techniques for handling complex time series analysis in a distributed computing environment.