Learn to use REST Proxy, Kafka Connect, KSQL, and Faust Python Stream Processing and use it to stream public transit statuses using Kafka and Kafka ecosystem to build a stream processing application that shows the status of trains in real-time.
Overview
Syllabus
- Introduction to Stream Processing
- In this lesson students will learn what data streaming is. Students will learn the pros and cons of data streaming, and how it compares to traditional data strategies.
- Apache Kafka
- In this lesson we’ll review the architecture and configuration of Apache Kafka.
- Data Schemas and Apache Avro
- This lesson covers data schemas and data schema management, with a focus on Apache Avro.
- Kafka Connect and REST Proxy
- This lesson covers producing and consuming data into Kafka with Kafka Connect and REST Proxy.
- Stream Processing Fundamentals
- Learn to build real-time applications that instantly process events, the concepts of stream processing state storage, windowed processing, and stateful and non-stateful stream processing.
- Stream Processing with Faust
- Students will learn how to use the Python stream processing library Faust to rapidly create powerful stream processing applications.
- KSQL
- Learn how to write simple SQL queries to turn Kafka topics into KSQL streams and tables, and then write those tables back out to Kafka.
- Optimizing Public Transportation
- For your first project, you’ll be streaming public transit status using Kafka and the Kafka ecosystem to build a stream processing application that shows the status of trains in real-time.
Taught by
Ben Goldberg
