Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Realtime Streaming with Apache Flink - End-to-End Data Engineering Project

CodeWithYu via YouTube

Overview

Build an end-to-end data engineering project using Apache Flink, Kafka, Elasticsearch, and Docker in this comprehensive video tutorial. Dive deep into real-time data processing and analytics, learning to create a robust, scalable data pipeline. Follow step-by-step instructions to set up the system architecture, generate sales analytics data, produce data into Kafka broker, configure Apache Flink, consume data from Kafka, write streams to PostgreSQL, aggregate transactions, and create real-time dashboards in Elasticsearch. Gain hands-on experience with industry-leading technologies and develop practical skills in big data engineering, streaming data processing, and real-time analytics.

Syllabus

Introduction
The system architecture
Sales Analytics Data Generation
Producing Data into Kafka Broker
Setting up Apache Flink project
Consuming data from Kafka with Apache Flink
Starting Apache Flink on Mac
Writing Kafka Streams to Postgres Database
Aggregating Transactions per Category into Postgres
Aggregating Transactions Per Day into Postgres
Aggregating Transactions Per Month into Postgres
Writing Kafka Streams Data into Elasticsearch
Reindexing Data on Elasticsearch with Timestamp
Creating Streaming Dashboard on Elasticsearch
Realtime Dashboard Results
Recap
Outro

Taught by

CodeWithYu

Reviews

Start your review of Realtime Streaming with Apache Flink - End-to-End Data Engineering Project

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.