Building Modern Data Streaming Apps with Open Source

Explore best practices for building modern data streaming applications using open-source frameworks in this comprehensive conference talk. Discover a cutting-edge approach that leverages Apache NiFi as the orchestrator for streams flowing into Apache Pulsar. Learn how to construct streaming ETL with Apache Spark and enhance events using Pulsar Functions for machine learning and enrichment. Dive into continuous queries against topics with Flink SQL and explore streaming data into various open-source data stores, including Apache Iceberg and Apache Pinot. Gain insights from seven years of experience in developing data streaming applications for IoT, CDC, logs, and more. Witness live coding demonstrations based on audience feedback, featuring new data stores, sources, and data relevant to the Vancouver area. Get up-to-date information on platform enhancements and the integration of emerging technologies in the open-source stack known as FLiPN.