Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Realtime Data Streaming - End-to-End Data Engineering Project

CodeWithYu via YouTube

Overview

Build a real-time data streaming pipeline in this comprehensive end-to-end data engineering project video. Learn to set up a data pipeline with Apache Airflow, stream data using Kafka and Kafka Connect, utilize Zookeeper for distributed synchronization, process data with Apache Spark, and implement data storage solutions with Cassandra and PostgreSQL. Master containerization of your data engineering environment using Docker. Follow along as the instructor guides you through each phase, from data ingestion to processing and storage, using a powerful stack of tools and technologies. Gain hands-on experience with system architecture, API data retrieval, Docker Compose setup, and streaming data into Kafka and Cassandra. Perfect for aspiring data engineers and professionals looking to enhance their skills in building scalable, real-time data processing systems.

Syllabus

Introduction
System architecture
Getting data from API with Airflow
Docker Compose for the architecture
Streaming data into Kafka
Apache Spark and Cassandra setup
Streaming data into cassandra
Outro

Taught by

CodeWithYu

Reviews

Start your review of Realtime Data Streaming - End-to-End Data Engineering Project

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.