Build a real estate end-to-end data engineering streaming pipeline in this comprehensive 2-hour video tutorial. Learn to gather, ingest, process, and store data using cutting-edge tools like Large Language Models (ChatGPT), WebSocket, Chrome DevTools Protocol, Docker, Apache Kafka, Apache Spark with Master-Worker Architecture, Apache Zookeeper, Confluent Control Center, and Cassandra. Follow along as the instructor guides you through setting up the project, automating manual processes, extracting property details with AI, configuring streaming architecture, and deploying the application to a Master-Worker cluster. Gain hands-on experience in real-time data streaming, ETL pipelines, and big data technologies while working with real estate data. Perfect for aspiring data engineers and professionals looking to enhance their skills in building scalable, real-time data processing systems.
Overview
Syllabus
Introduction
The system architecture
Getting your keys
Setting up a new project
Installing the required dependencies
Manual Simulation of the Process
Automating the Manual Process
Using Large Language Models to Extract Property Details
Setting up Streaming Architecture
Streaming Property information to Kafka
Consuming Realtime streams from Kafka
Writing Property Data to Cassandra Storage
Deploying Streaming App to Master-Worker Cluster
Fixing scala/$less$colon$less bug with Spark Streaming
Verification of Results
End to End Process Automation
Outro
Taught by
CodeWithYu
