Processing and Storing Time Series Instrument Data with Pandas and PySpark

Learn how to effectively process and store time series instrument data in this 29-minute conference talk from PyBay. Discover practical solutions for handling scientific data from fermentation, environmental sensors, and spectroscopy instruments that often come in proprietary or unusual formats. Explore how to leverage pandas for processing uniquely formatted files and pyspark through Databricks for managing large-scale datasets. Master techniques for transforming complex laboratory data from traditional transactional, row-oriented database systems into more analyzable formats. Presented by Aaron Wiegel at SF Python's PyBay conference, gain insights into overcoming common challenges faced when dealing with enterprise laboratory information management systems (LIMS) and processing millions of records efficiently.