Overview
Syllabus
intro
preamble
about chinmay naik
mongodb to rdbms data migration
student collection mongodb
student table postresql
student - address and phone relationships
data migration - mongodb to postresql
how mongodb json data maps to sql
inserts are cool, what about updates and deletes in mongodb?
how do we migrate data?
mongo oplog operation log
what does oplog record look like?
when are we gerring to the golang concurrency?
sequential data pipeline
mongo oplog / two oplogs / postgresql
sequential pipeline performance
perf improvemwent - let's add worker pool
worker pool
worker pools v2.0
worker pool v2.0 performance
can you guess the problem?
worker pools v2.0 - the problem
back to drawing board?
fan-out for each database
concurrent data pipeline
performance comparison
resource utilization
concurrent data pipeline - improvement
16 databases and 128 collections per db
performance comparison
final concurrent data pipeline
key takeaways
keep learning
Taught by
Conf42