Explore how to track database changes efficiently using Apache Kafka in this 54-minute conference talk. Learn about the limitations of traditional ETL flows and discover a more performant alternative using Kafka Connect. Understand the basics of Apache Kafka, its cluster architecture, and how it integrates with existing data platforms like PostgreSQL. Dive into various Kafka Connect options, including timestamp-based and log-based approaches, and their respective benefits and limitations. Gain insights on configuring Kafka Connect, implementing the Division connector, and simplifying your data pipeline. By the end of this talk, acquire the knowledge to avoid "Spaghetti architecture" and leverage Apache Kafka as the foundation for a streamlined, high-performance data streaming solution.
I Don't Want to Miss a Thing - Track Database Changes with Apache Kafka
Overview
Syllabus
Introduction
Example
Lessons
What is Kafka
Kafka Clusters
The Fridge Dilemma
Kafka Connect
Database Changes
Timestamp Approach
Kafka Connect Configuration
Limitations of querybased approach
Logbased approach
Division connector
Division connector configuration
Review
Taught by
NDC Conferences
Related Courses
-
Streaming ETL on the Shoulders of Giants
-
Streaming Database Changes with Debezium
-
Big Data Analytics in Near-Real-Time with Apache Kafka Streams
-
Calculating the Value of Pie - Real-Time Survey Analysis With Apache Kafka
-
Building Event-Driven Microservices with Apache Kafka
-
The Database Unbundled - Commit Logs in an Age of Microservices
