Processing Data at Scale with Rails

Explore strategies for processing large-scale data using Ruby on Rails in this 35-minute conference talk. Learn how to handle massive datasets from various sources, including public records, shared resources, and IoT data. Discover techniques for ingesting, transforming, and storing data efficiently using familiar tools like ActiveRecord, Sidekiq, and Postgres. Gain insights into breaking down large data jobs into manageable tasks, turning data firehoses into structured pipelines. Dive into topics such as ETL processes, pipeline options, and practical examples using lobby data. Understand the importance of scraping, Redis usage, fanning out processes, load jobs, unique indexing, and data normalization. Address scaling concerns and explore opportunities for optimizing data processing in Rails applications.