Embark on a comprehensive end-to-end data engineering project to build a real-time voting system using modern big data technologies such as Apache Kafka, Apache Spark, and Streamlit. Learn to design and implement the election flow, set up the system architecture, and develop key components including candidate and voter accreditation, vote streaming, data preprocessing, and aggregation. Create a real-time dashboard to visualize election results and live commentaries. Gain hands-on experience with Docker, PySpark, and Streamlit while exploring the intricacies of building a robust, scalable voting system for modern elections.
Overview
Syllabus
Introduction
The Election Flow
The System Architecture
Setting up the Project
The docker compose for the system architecture
Candidate, Party and Voters Accredditation
Voting System
Streaming votes from/to Kafka
Reading Voting Data Streams from Kafka with Pyspark
Spark Data preprocessing
Spark Aggregations
Writing aggregated data back to kafka
Realtime Votes Dashboard with Streamlit
Results and Live commentaries
Winner of the Election
31 Outro
Taught by
CodeWithYu
Related Courses
-
Data Streaming
3.5 -
Applying the Lambda Architecture with Spark, Kafka, and Cassandra
-
Streaming Big Data with Spark Streaming, Scala, and Spark 3!
-
Realtime Data Streaming - End-to-End Data Engineering Project
-
Realtime Streaming with Apache Flink - End-to-End Data Engineering Project
-
Realtime Socket Streaming with Apache Spark - End-to-End Data Engineering Project
