Build an end-to-end data engineering project using Apache Flink, Kafka, Elasticsearch, and Docker in this comprehensive video tutorial. Dive deep into real-time data processing and analytics, learning to create a robust, scalable data pipeline. Follow step-by-step instructions to set up the system architecture, generate sales analytics data, produce data into Kafka broker, configure Apache Flink, consume data from Kafka, write streams to PostgreSQL, aggregate transactions, and create real-time dashboards in Elasticsearch. Gain hands-on experience with industry-leading technologies and develop practical skills in big data engineering, streaming data processing, and real-time analytics.
Overview
Syllabus
Introduction
The system architecture
Sales Analytics Data Generation
Producing Data into Kafka Broker
Setting up Apache Flink project
Consuming data from Kafka with Apache Flink
Starting Apache Flink on Mac
Writing Kafka Streams to Postgres Database
Aggregating Transactions per Category into Postgres
Aggregating Transactions Per Day into Postgres
Aggregating Transactions Per Month into Postgres
Writing Kafka Streams Data into Elasticsearch
Reindexing Data on Elasticsearch with Timestamp
Creating Streaming Dashboard on Elasticsearch
Realtime Dashboard Results
Recap
Outro
Taught by
CodeWithYu
