Streaming Data Pipelines on Apache Mesos - Lessons Learned

Explore streaming data pipelines on Apache Mesos in this 42-minute conference talk by Dean Wampler, Ph.D. from Lightbend. Gain insights into the Lightbend Fast Data Platform, a suite of open-source streaming tools running on Mesosphere DC/OS. Discover why Mesos was chosen for its flexibility in supporting diverse requirements of streaming and batch data processing. Learn about the key Mesos features crucial for Lightbend's needs, illustrated through examples of how tools like Kafka, Spark, Flink, Akka Streams, and Kafka Streams work with Mesos. Delve into lessons learned from running applications on Mesos and get a glimpse of future developments. Cover topics including Hadoop's limitations, streaming services, microservices, Spark Streaming, continuous processing, scalability, checkpointing, dynamic resource allocation, machine learning, interoperability, and various streaming tools. Understand the relationship between microservices and Fast Data, exploring common mechanisms and architectures in this informative talk by a big data expert.