Sound Data Engineering in Rust - From Bits to DataFrames

Explore sound data engineering principles in Rust, from fundamental bits to advanced DataFrames, in this 35-minute Databricks conference talk. Dive into Spark's Data Source APIs and their optimization techniques for querying external data sources. Learn about filter push down, column pruning, and the newly introduced partial aggregate push down, which significantly improves query performance. Discover how these optimizations are implemented in JDBC and Parquet. Examine the relationship between information storage and usage, CPU utilization, and the role of Apache Arrow in data processing. Gain insights into the Rust programming language and its integration with Arrow. Watch a demo showcasing practical applications, explore who uses arrow2, and understand the benefits of Polars. Analyze benchmarks and key takeaways from the DATA+AI SUMMIT 2022 presentation on efficient data engineering practices.