Speed is Not All You Need for Data Processing

Explore the evolving landscape of data processing frameworks in this PyCon US talk by Kevin Kho and Han Wang. Delve into the comparison between traditional tools like Pandas and newer options such as Polars and DuckDB. Examine the common assumptions about speed and performance in data processing, particularly focusing on Pandas and Pandas UDFs in relation to native Spark. Through benchmarks across various data sizes, discover surprising insights that challenge conventional wisdom about tool selection. Learn why practitioners should prioritize tools that best serve their needs rather than solely focusing on perceived speed advantages. Gain valuable perspective on making informed decisions when choosing data processing frameworks for your projects.