Swiss Transport in Real Time - Tribulations in the Big Data Stack

Explore a proof-of-concept project utilizing real-time Swiss public transportation data to build a scalable infrastructure for dispatching, transforming, and visualizing massive data streams. Learn how to address the challenge of creating a simple, scalable system for near real-time data processing and subsequent analysis. Dive into the various components of the infrastructure, including streaming events with Kafka and Logstash, flow transformation using Akka or Play Streaming, and data storage in Elasticsearch. Discover techniques for real-time visualization with ReactJS and d3.js, as well as a posteriori analysis using Python and Jupyter. Gain insights into DevOps practices with Docker, Google Cloud Engine, and Amazon Web Services. This conference talk provides a comprehensive overview of building a big data stack for handling real-time transportation data, offering valuable lessons for developers and data engineers working on similar large-scale data projects.